Applying patch to Exadata

Hi everyone! I’m here to post about the patch apply on Exadata Machine. As best practices we will apply the QFSP (Quatterly Full Stack Patch) for Exadata Jan/2014. The patch apply is totally automatic so if the prereqs were addressed correctly, you will have no bad surprise and your Exadata environment will be patched successfully. At my job, our team applied it recently without any issue.

The patch number is 17816100 [Quarterly Full Stack Download Patch For Oracle Exadata (Jan 2014 – 11.2.3.3.0)] which has 3.6G . This patch will patch most of the Exadata Database Machine components, whic are: databases; dbnodes; storage servers; infiniband switches; and PDUs (Power Distribution Units). Our databases are already patched to version (11.2.0.3.21) and on the end of this patching, the image version for the db and cell nodes should be 11.2.3.3.0 as we are moving from image 11.2.3.2.1.

You should carefully read all the README and notes regarding this patch as there is a complete list of prereqs and things to analyze. Although the db and cell nodes will all end with the same image version, on our case, the infiniband switches upgrade was optional according to the compatibility matrix but to keep things easy, we upgraded them too. The PDUs upgrade are optional and these is the easiest one.

Now lets get hands on it and lets begin with the PDUs. Doing this upgrade will cost you no outage and it is as simple as upgrading the firmware from your home network router. Just navigate to your PDU from your browser and hit “Net Configuration”. Scroll down to “Firmware Upgrade” and select the file MKAPP_Vx.x.dl to upgrade. After the PDU firmware was upgraded it will popup for the HTML interface to be upgraded so you need to select the file HTML_Vx.x.dl. Do that on all of the PDUs and your are done with it. Peace of cake.

Now lets proceed to the cells upgrade. As we usage the rolling upgrade strategy (no outage), all of the database software must have 17854520 patch applied on them, other while, the DBs may hang or crash. The utility used to patch the cells and infiniband switches is patchmgr (which should be executed as root). Also, you can run a precheck for the upgrade from this utility, as mentioned below:

# ./patchmgr -cells cell_group -patch_check_prereq -rolling

It is recommended to higher the disk repair time from diskgroups, in order to do not drop the disks. Also, and according to Oracle docs, it is recommended to reset the cells if this is the first time that those cells image are upgraded. Do this one cell at a time and then initiate the cell upgrade. The patchmgr should be executed from the dbnode.

# ./patchmgr -cells cel01 -reset_force
# ./patchmgr -cells cel02 -reset_force
# ./patchmgr -cells cel03 -reset_force
# ./patchmgr -cells cell_group -rolling

After finishing successfully the cells upgrade, go for infiniband switches precheck upgrade and execute the patchmgr utility as listed below:

# ./patchmgr -ibswitches -upgrade -ibswitch_precheck

To continue with the ib switches upgrade just remove the precheck parameter:

# ./patchmgr -ibswitches -upgrade

When you are done with the infiniband switches and the cell nodes you should go to upgrade the database nodes. For this upgrade, you will have the dbnodeupdate.sh utility. This will upgrade dbnodes kernel and all of the dependent packages. Pay attention that if you have any other third package installed you should upgrade it manually after the upgrade. On our environment, the kernel will be upgrade to Oracle Linux 5.9 (kernel-2.6.39-400.126.1.el5uek).The dbnodeupdate.sh is fully automatic and it will disable and bring down the CRS for the node. You must use root user to run it and for best practices do it one node at a time.

To perform a precheck run it with the parameter -v on the end
# ./dbnodeupdate.sh -u -l $PATCH_17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/p17809253_112330_Linux-x86-64.zip -v

Now to start the upgrade for the dbnode, execute it without the -v parameter
# ./dbnodeupdate.sh -u -l $PATCH_17816100/Infrastructure/ExadataStorageServer/11.2.3.3.0/p17809253_112330_Linux-x86-64.zip

After the machine reboots, confirm the upgrade executing:
# ./dbnodeupdate.sh -c

Perform this steps on all the dbnodes remaining and you are done. The whole Exadata Machine is patched, run imageinfo on all dbnodes e storage servers to confirm the new image. On the ibswitches run the command version to confirm it:

# dcli -g all_group -l root imageinfo
db01:
db01: Kernel version: 2.6.39-400.126.1.el5uek #1 SMP Fri Sep 20 10:54:38 PDT 2013 x86_64
db01: Image version: 11.2.3.3.0.131014.1
db01: Image activated: 2014-03-29 10:30:56 -0300
db01: Image status: success
db01: System partition on device: /dev/mapper/VGExaDb-LVDbSys1
db01:

db02:
db02: Kernel version: 2.6.39-400.126.1.el5uek #1 SMP Fri Sep 20 10:54:38 PDT 2013 x86_64
db02: Image version: 11.2.3.3.0.131014.1
db02: Image activated: 2014-03-30 10:23:58 -0300
db02: Image status: success
db02: System partition on device: /dev/mapper/VGExaDb-LVDbSys1
db02:

cel01:
cel01: Kernel version: 2.6.39-400.126.1.el5uek #1 SMP Fri Sep 20 10:54:38 PDT 2013 x86_64
cel01: Cell version: OSS_11.2.3.3.0_LINUX.X64_131014.1
cel01: Cell rpm version: cell-11.2.3.3.0_LINUX.X64_131014.1-1
cel01:
cel01: Active image version: 11.2.3.3.0.131014.1
cel01: Active image activated: 2014-03-28 23:42:33 -0300
cel01: Active image status: success
cel01: Active system partition on device: /dev/md6
cel01: Active software partition on device: /dev/md8
cel01:
cel01: In partition rollback: Impossible
cel01:
cel01: Cell boot usb partition: /dev/sdm1
cel01: Cell boot usb version: 11.2.3.3.0.131014.1
cel01:
cel01: Inactive image version: 11.2.3.1.0.120304
cel01: Inactive image activated: 2012-05-21 18:00:09 -0300
cel01: Inactive image status: success
cel01: Inactive system partition on device: /dev/md5
cel01: Inactive software partition on device: /dev/md7
cel01:
cel01: Boot area has rollback archive for the version: 11.2.3.1.0.120304
cel01: Rollback to the inactive partitions: Possible

cel02:
cel02: Kernel version: 2.6.39-400.126.1.el5uek #1 SMP Fri Sep 20 10:54:38 PDT 2013 x86_64
cel02: Cell version: OSS_11.2.3.3.0_LINUX.X64_131014.1
cel02: Cell rpm version: cell-11.2.3.3.0_LINUX.X64_131014.1-1
cel02:
cel02: Active image version: 11.2.3.3.0.131014.1
cel02: Active image activated: 2014-03-29 00:46:13 -0300
cel02: Active image status: success
cel02: Active system partition on device: /dev/md6
cel02: Active software partition on device: /dev/md8
cel02:
cel02: In partition rollback: Impossible
cel02:
cel02: Cell boot usb partition: /dev/sdm1
cel02: Cell boot usb version: 11.2.3.3.0.131014.1
cel02:
cel02: Inactive image version: 11.2.3.1.0.120304
cel02: Inactive image activated: 2012-05-21 18:01:07 -0300
cel02: Inactive image status: success
cel02: Inactive system partition on device: /dev/md5
cel02: Inactive software partition on device: /dev/md7
cel02:
cel02: Boot area has rollback archive for the version: 11.2.3.1.0.120304
cel02: Rollback to the inactive partitions: Possible

cel03:
cel03: Kernel version: 2.6.39-400.126.1.el5uek #1 SMP Fri Sep 20 10:54:38 PDT 2013 x86_64
cel03: Cell version: OSS_11.2.3.3.0_LINUX.X64_131014.1
cel03: Cell rpm version: cell-11.2.3.3.0_LINUX.X64_131014.1-1
cel03:
cel03: Active image version: 11.2.3.3.0.131014.1
cel03: Active image activated: 2014-03-29 01:51:22 -0300
cel03: Active image status: success
cel03: Active system partition on device: /dev/md6
cel03: Active software partition on device: /dev/md8
cel03:
cel03: In partition rollback: Impossible
cel03:
cel03: Cell boot usb partition: /dev/sdm1
cel03: Cell boot usb version: 11.2.3.3.0.131014.1
cel03:
cel03: Inactive image version: 11.2.3.1.0.120304
cel03: Inactive image activated: 2012-05-21 18:01:28 -0300
cel03: Inactive image status: success
cel03: Inactive system partition on device: /dev/md5
cel03: Inactive software partition on device: /dev/md7
cel03:
cel03: Boot area has rollback archive for the version: 11.2.3.1.0.120304
cel03: Rollback to the inactive partitions: Possible

sw-ib2 # version
SUN DCS 36p version: 2.1.3-4
Build time: Aug 28 2013 16:25:57
SP board info:
Manufacturing Date: 2011.05.08
Serial Number: “NCD6I0106”
Hardware Revision: 0x0006
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010

sw-ib3 # version
SUN DCS 36p version: 2.1.3-4
Build time: Aug 28 2013 16:25:57
SP board info:
Manufacturing Date: 2011.05.11
Serial Number: “NCD6Q0110”
Hardware Revision: 0x0006
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010

Docs:

• Exadata 11.2.3.3.0 release and patch (16278923) (Doc ID 1487339.1)
• Exadata Database Server Patching using the DB Node Update Utility (Doc ID 1553103.1)
• Exadata Patching Overview and Patch Testing Guidelines (Doc ID 1262380.1)
• Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1)

Thats it guys!

3 thoughts on “Applying patch to Exadata

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s