Patching Exadata QFSP July 2015

Hey, everyone! I’m here with this shortly post about patching Exadata QFSP July 2015. My teammate and I have recently patched our X2-2 Half Rack environment from 11.2.3.3.0.131014.1 to 12.1.2.1.2.150617.1 so I want to THANK them (Vitor Eduardo, Claudio Angerami, Bruno Palma, Anselmo Ribeiro and Edmilson Carmo) for the great job we’ve done. There are no big news, nothing really changed from the other post that I made before, the big key is to pay attention on the ‘Known Issues’ and address them as founded. Also, analyze the RPMs that will be deleted in order to guarantee same functionality as before. After that, if everything is fine your platform should be ready to patch.

So let’s go for it! .Just a point here, we changed the real hostnames and IPs from the servers, cells and switches.

  • IB SWITCHES UPGRADE:

[root@exa01db01 patch_12.1.2.1.2.150617.1]# ./patchmgr -ibswitches /root/ib_switches -upgrade

With arguments: -ibswitches /root/ib_switches -upgrade
2015-08-08 17:01:37 -0300 [INFO] pid file: /var/log/exadatatmp/_EXA_AXE_patchmgr.lock : 98220
PID TTY TIME CMD
2015-08-08 17:01:42 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
2015-08-08 17:01:42 -0300 ++++++++++++++++++ Logs so far end ++++++++++
2015-08-08 17:01:42 -0300 1 of 1 :SUCCESS: DO: Initiate upgrade of InfiniBand switches to 2.1.5-1. Expect up to 15 minutes for each switch
2015-08-08 17:45:23 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
—– InfiniBand switch update process started Sat Aug 8 17:01:42 BRT 2015 —–
[NOTE ] Log file at /var/log/cellos/upgradeIBSwitch.log

[INFO ] List of InfiniBand switches for upgrade: ( exa01sw-ib2 exa01sw-ib3 )
[PROMPT ] Use the default password for all switches? (y/n) [n]:
[PROMPT ] Updating only 2 switch(es). Are you sure you want to continue? (y/n) [n]:
[SUCCESS ] Verifying Network connectivity to exa01sw-ib2
[SUCCESS ] Verifying Network connectivity to exa01sw-ib3
[SUCCESS ] Validating verify-topology output
[INFO ] Proceeding with upgrade of InfiniBand switches to version 2.1.5_1
[INFO ] Master Subnet Manager is set to “exa01sw-ib2” in all Switches

[INFO ] ———- Starting with IBSwitch exa01sw-ib2
[SUCCESS ] Disable Subnet Manager on exa01sw-ib2
[SUCCESS ] Copy firmware packages to exa01sw-ib2
[SUCCESS ] exa01sw-ib2 is at 2.1.3-4. Meets minimal patching level 2.1.3-4
[SUCCESS ] Verifying that /tmp has 120M in exa01sw-ib2, found 138M
[SUCCESS ] Verifying that / has 80M in exa01sw-ib2, found 198M
[SUCCESS ] Verifying that exa01sw-ib2 has 120M free memory, found 239M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth[0,1] for exa01sw-ib2
[SUCCESS ] Verifying that exa01sw-ib2 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-08-08 Time:(HH:MM:SS) 17:03:03
[SUCCESS ] Execute plugin check for Patch Check Prereq on exa01sw-ib2
[SUCCESS ] Pre-update validation on exa01sw-ib2
[INFO ] Starting upgrade on exa01sw-ib2 to 2.1.5_1. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[SUCCESS ] Execute plugin check for Patching on exa01sw-ib2
[SUCCESS ] Load firmware 2.1.5_1 onto exa01sw-ib2
[SUCCESS ] Disable Subnet Manager on exa01sw-ib2
[SUCCESS ] Verify that /conf/configvalid is set to 1 on exa01sw-ib2
[SUCCESS ] Set SMPriority to 5 on exa01sw-ib2
[INFO ] Rebooting exa01sw-ib2. Wait for 240 secs before continuing
[SUCCESS ] Reboot exa01sw-ib2
[SUCCESS ] Restart Subnet Manager on exa01sw-ib2
[INFO ] Starting post-update validation on exa01sw-ib2
[SUCCESS ] Inifiniband switch exa01sw-ib2 is at target patching level
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth[0,1] for exa01sw-ib2
[SUCCESS ] Verifying that exa01sw-ib2 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-08-08 Time:(HH:MM:SS) 17:19:42
[SUCCESS ] Firmware verification on InfiniBand switch exa01sw-ib2
[SUCCESS ] Execute plugin check for Post Patch on exa01sw-ib2
[SUCCESS ] Post-check validation on IBSwitch exa01sw-ib2
[SUCCESS ] Update switch exa01sw-ib2 to 2.1.5_1

[INFO ] ———- Starting with InfiniBand Switch exa01sw-ib3
[SUCCESS ] Disable Subnet Manager on exa01sw-ib3
[SUCCESS ] Copy firmware packages to exa01sw-ib3
[SUCCESS ] exa01sw-ib3 is at 2.1.3-4. Meets minimal patching level 2.1.3-4
[SUCCESS ] Verifying that /tmp has 120M in exa01sw-ib3, found 138M
[SUCCESS ] Verifying that / has 80M in exa01sw-ib3, found 199M
[SUCCESS ] Verifying that exa01sw-ib3 has 120M free memory, found 237M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth[0,1] for exa01sw-ib3
[SUCCESS ] Verifying that exa01sw-ib3 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-08-08 Time:(HH:MM:SS) 17:24:40
[SUCCESS ] Execute plugin check for Patch Check Prereq on exa01sw-ib3
[SUCCESS ] Pre-update validation on exa01sw-ib3
[INFO ] Starting upgrade on exa01sw-ib3 to 2.1.5_1. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[SUCCESS ] Execute plugin check for Patching on exa01sw-ib3
[SUCCESS ] Load firmware 2.1.5_1 onto exa01sw-ib3
[SUCCESS ] Disable Subnet Manager on exa01sw-ib3
[SUCCESS ] Verify that /conf/configvalid is set to 1 on exa01sw-ib3
[SUCCESS ] Set SMPriority to 5 on exa01sw-ib3
[INFO ] Rebooting exa01sw-ib3. Wait for 240 secs before continuing
[SUCCESS ] Reboot exa01sw-ib3
[SUCCESS ] Restart Subnet Manager on exa01sw-ib3
[INFO ] Starting post-update validation on exa01sw-ib3
[SUCCESS ] Inifiniband switch exa01sw-ib3 is at target patching level
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth[0,1] for exa01sw-ib3
[SUCCESS ] Verifying that exa01sw-ib3 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-08-08 Time:(HH:MM:SS) 17:41:09
[SUCCESS ] Firmware verification on InfiniBand switch exa01sw-ib3
[SUCCESS ] Execute plugin check for Post Patch on exa01sw-ib3
[SUCCESS ] Post-check validation on IBSwitch exa01sw-ib3
[SUCCESS ] Update switch exa01sw-ib3 to 2.1.5_1
[INFO ] InfiniBand Switches ( exa01sw-ib2 exa01sw-ib3 ) updated to 2.1.5_1
[SUCCESS ] Overall status

—– InfiniBand switch update process ended Sat Aug 8 17:45:23 BRT 2015 —–
2015-08-08 17:45:23 -0300 ++++++++++++++++++ Logs so far end ++++++++++
2015-08-08 17:45:23 -0300 1 of 1 :SUCCESS: DONE: Upgrade InfiniBand switch(es) to 2.1.5-1.
================PatchMgr run ended 2015-08-08 17:45:23 -0300 ===========


  • CELL NODES UPGRADE:

For this one, you need to execute -reset_force and -cleanup procedures before upgrade, as below:

[root@ndcing01db01 patch_12.1.2.1.2.150617.1]# ./patchmgr -cells /root/cell_group -reset_force
================PatchMgr run started 2015-08-08 00:21:13 -0300 ===========
With arguments: -cells /root/cell_group -reset_force
2015-08-08 00:21:13 -0300 [INFO] pid file: /var/log/exadatatmp/_EXA_AXE_patchmgr.lock : 72003
PID TTY TIME CMD
[INFO] Reset force was successful.
2015-08-08 00:21:18 -0300 :DONE: reset_force
================PatchMgr run ended 2015-08-08 00:21:18 -0300 ===========

[root@ndcing01db01 patch_12.1.2.1.2.150617.1]# ./patchmgr -cells /root/cell_group -cleanup
================PatchMgr run started 2015-08-08 00:21:45 -0300 ===========
With arguments: -cells /root/cell_group -cleanup
2015-08-08 00:22:50 -0300 :Working: DO: Cleanup …
2015-08-08 00:22:51 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
[INFO] Reset force was successful.
2015-08-08 00:22:51 -0300 ++++++++++++++++++ Logs so far end ++++++++++
2015-08-08 00:22:51 -0300 :SUCCESS: DONE: Cleanup
================PatchMgr run ended 2015-08-08 00:22:51 -0300 ===========

After that, you are ready to patch the cells in a rolling fashion:

[root@ndcing01db01 patch_12.1.2.1.2.150617.1]# ./patchmgr -cells /root/cell_group -patch -rolling
================PatchMgr run started 2015-08-08 00:26:02 -0300 ===========
With arguments: -cells /root/cell_group -patch -rolling
2015-08-08 00:27:07 -0300 :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell …
2015-08-08 00:27:07 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
2015-08-08 00:27:08 -0300 ++++++++++++++++++ Logs so far end ++++++++++
2015-08-08 00:27:08 -0300 :SUCCESS: DONE: Check cells have ssh equivalence for root user.
2015-08-08 00:27:11 -0300 :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute …
2015-08-08 00:27:28 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
.
.
.

I didn’t posted the whole log as this is a big one, but it is here so feel free to take a look at it. And finally we went to database nodes.

  • DB NODES UPGRADE:

[root@exa01db01 5.150701]# ./dbnodeupdate.sh -u -l /u01/install/quarterly_full_jul2015/21339383/Infrastructure/12.1.2.1.2/ExadataDatabaseServer_OL6/p21151982_121212_Linux-x86-64.zip -s

After this procedure thee server get reboot and it takes a while to bring up. If everything is fine with the patching, complete the patching:

[root@exa01db01 5.150701]# ./dbnodeupdate.sh -c

That’s it guys, upgrade finished successfully, everything went smoothly. Hope you enjoyed it!