Patching Exadata QFSP July 2015

Hey, everyone! I’m here with this shortly post about patching Exadata QFSP July 2015. My teammate and I have recently patched our X2-2 Half Rack environment from 11.2.3.3.0.131014.1 to 12.1.2.1.2.150617.1 so I want to THANK them (Vitor Eduardo, Claudio Angerami, Bruno Palma, Anselmo Ribeiro and Edmilson Carmo) for the great job we’ve done. There are no big news, nothing really changed from the other post that I made before, the big key is to pay attention on the ‘Known Issues’ and address them as founded. Also, analyze the RPMs that will be deleted in order to guarantee same functionality as before. After that, if everything is fine your platform should be ready to patch.

So let’s go for it! .Just a point here, we changed the real hostnames and IPs from the servers, cells and switches.

  • IB SWITCHES UPGRADE:

[root@exa01db01 patch_12.1.2.1.2.150617.1]# ./patchmgr -ibswitches /root/ib_switches -upgrade

With arguments: -ibswitches /root/ib_switches -upgrade
2015-08-08 17:01:37 -0300 [INFO] pid file: /var/log/exadatatmp/_EXA_AXE_patchmgr.lock : 98220
PID TTY TIME CMD
2015-08-08 17:01:42 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
2015-08-08 17:01:42 -0300 ++++++++++++++++++ Logs so far end ++++++++++
2015-08-08 17:01:42 -0300 1 of 1 :SUCCESS: DO: Initiate upgrade of InfiniBand switches to 2.1.5-1. Expect up to 15 minutes for each switch
2015-08-08 17:45:23 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
—– InfiniBand switch update process started Sat Aug 8 17:01:42 BRT 2015 —–
[NOTE ] Log file at /var/log/cellos/upgradeIBSwitch.log

[INFO ] List of InfiniBand switches for upgrade: ( exa01sw-ib2 exa01sw-ib3 )
[PROMPT ] Use the default password for all switches? (y/n) [n]:
[PROMPT ] Updating only 2 switch(es). Are you sure you want to continue? (y/n) [n]:
[SUCCESS ] Verifying Network connectivity to exa01sw-ib2
[SUCCESS ] Verifying Network connectivity to exa01sw-ib3
[SUCCESS ] Validating verify-topology output
[INFO ] Proceeding with upgrade of InfiniBand switches to version 2.1.5_1
[INFO ] Master Subnet Manager is set to “exa01sw-ib2” in all Switches

[INFO ] ———- Starting with IBSwitch exa01sw-ib2
[SUCCESS ] Disable Subnet Manager on exa01sw-ib2
[SUCCESS ] Copy firmware packages to exa01sw-ib2
[SUCCESS ] exa01sw-ib2 is at 2.1.3-4. Meets minimal patching level 2.1.3-4
[SUCCESS ] Verifying that /tmp has 120M in exa01sw-ib2, found 138M
[SUCCESS ] Verifying that / has 80M in exa01sw-ib2, found 198M
[SUCCESS ] Verifying that exa01sw-ib2 has 120M free memory, found 239M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth[0,1] for exa01sw-ib2
[SUCCESS ] Verifying that exa01sw-ib2 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-08-08 Time:(HH:MM:SS) 17:03:03
[SUCCESS ] Execute plugin check for Patch Check Prereq on exa01sw-ib2
[SUCCESS ] Pre-update validation on exa01sw-ib2
[INFO ] Starting upgrade on exa01sw-ib2 to 2.1.5_1. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[SUCCESS ] Execute plugin check for Patching on exa01sw-ib2
[SUCCESS ] Load firmware 2.1.5_1 onto exa01sw-ib2
[SUCCESS ] Disable Subnet Manager on exa01sw-ib2
[SUCCESS ] Verify that /conf/configvalid is set to 1 on exa01sw-ib2
[SUCCESS ] Set SMPriority to 5 on exa01sw-ib2
[INFO ] Rebooting exa01sw-ib2. Wait for 240 secs before continuing
[SUCCESS ] Reboot exa01sw-ib2
[SUCCESS ] Restart Subnet Manager on exa01sw-ib2
[INFO ] Starting post-update validation on exa01sw-ib2
[SUCCESS ] Inifiniband switch exa01sw-ib2 is at target patching level
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth[0,1] for exa01sw-ib2
[SUCCESS ] Verifying that exa01sw-ib2 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-08-08 Time:(HH:MM:SS) 17:19:42
[SUCCESS ] Firmware verification on InfiniBand switch exa01sw-ib2
[SUCCESS ] Execute plugin check for Post Patch on exa01sw-ib2
[SUCCESS ] Post-check validation on IBSwitch exa01sw-ib2
[SUCCESS ] Update switch exa01sw-ib2 to 2.1.5_1

[INFO ] ———- Starting with InfiniBand Switch exa01sw-ib3
[SUCCESS ] Disable Subnet Manager on exa01sw-ib3
[SUCCESS ] Copy firmware packages to exa01sw-ib3
[SUCCESS ] exa01sw-ib3 is at 2.1.3-4. Meets minimal patching level 2.1.3-4
[SUCCESS ] Verifying that /tmp has 120M in exa01sw-ib3, found 138M
[SUCCESS ] Verifying that / has 80M in exa01sw-ib3, found 199M
[SUCCESS ] Verifying that exa01sw-ib3 has 120M free memory, found 237M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth[0,1] for exa01sw-ib3
[SUCCESS ] Verifying that exa01sw-ib3 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-08-08 Time:(HH:MM:SS) 17:24:40
[SUCCESS ] Execute plugin check for Patch Check Prereq on exa01sw-ib3
[SUCCESS ] Pre-update validation on exa01sw-ib3
[INFO ] Starting upgrade on exa01sw-ib3 to 2.1.5_1. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[SUCCESS ] Execute plugin check for Patching on exa01sw-ib3
[SUCCESS ] Load firmware 2.1.5_1 onto exa01sw-ib3
[SUCCESS ] Disable Subnet Manager on exa01sw-ib3
[SUCCESS ] Verify that /conf/configvalid is set to 1 on exa01sw-ib3
[SUCCESS ] Set SMPriority to 5 on exa01sw-ib3
[INFO ] Rebooting exa01sw-ib3. Wait for 240 secs before continuing
[SUCCESS ] Reboot exa01sw-ib3
[SUCCESS ] Restart Subnet Manager on exa01sw-ib3
[INFO ] Starting post-update validation on exa01sw-ib3
[SUCCESS ] Inifiniband switch exa01sw-ib3 is at target patching level
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth[0,1] for exa01sw-ib3
[SUCCESS ] Verifying that exa01sw-ib3 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-08-08 Time:(HH:MM:SS) 17:41:09
[SUCCESS ] Firmware verification on InfiniBand switch exa01sw-ib3
[SUCCESS ] Execute plugin check for Post Patch on exa01sw-ib3
[SUCCESS ] Post-check validation on IBSwitch exa01sw-ib3
[SUCCESS ] Update switch exa01sw-ib3 to 2.1.5_1
[INFO ] InfiniBand Switches ( exa01sw-ib2 exa01sw-ib3 ) updated to 2.1.5_1
[SUCCESS ] Overall status

—– InfiniBand switch update process ended Sat Aug 8 17:45:23 BRT 2015 —–
2015-08-08 17:45:23 -0300 ++++++++++++++++++ Logs so far end ++++++++++
2015-08-08 17:45:23 -0300 1 of 1 :SUCCESS: DONE: Upgrade InfiniBand switch(es) to 2.1.5-1.
================PatchMgr run ended 2015-08-08 17:45:23 -0300 ===========


  • CELL NODES UPGRADE:

For this one, you need to execute -reset_force and -cleanup procedures before upgrade, as below:

[root@ndcing01db01 patch_12.1.2.1.2.150617.1]# ./patchmgr -cells /root/cell_group -reset_force
================PatchMgr run started 2015-08-08 00:21:13 -0300 ===========
With arguments: -cells /root/cell_group -reset_force
2015-08-08 00:21:13 -0300 [INFO] pid file: /var/log/exadatatmp/_EXA_AXE_patchmgr.lock : 72003
PID TTY TIME CMD
[INFO] Reset force was successful.
2015-08-08 00:21:18 -0300 :DONE: reset_force
================PatchMgr run ended 2015-08-08 00:21:18 -0300 ===========

[root@ndcing01db01 patch_12.1.2.1.2.150617.1]# ./patchmgr -cells /root/cell_group -cleanup
================PatchMgr run started 2015-08-08 00:21:45 -0300 ===========
With arguments: -cells /root/cell_group -cleanup
2015-08-08 00:22:50 -0300 :Working: DO: Cleanup …
2015-08-08 00:22:51 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
[INFO] Reset force was successful.
2015-08-08 00:22:51 -0300 ++++++++++++++++++ Logs so far end ++++++++++
2015-08-08 00:22:51 -0300 :SUCCESS: DONE: Cleanup
================PatchMgr run ended 2015-08-08 00:22:51 -0300 ===========

After that, you are ready to patch the cells in a rolling fashion:

[root@ndcing01db01 patch_12.1.2.1.2.150617.1]# ./patchmgr -cells /root/cell_group -patch -rolling
================PatchMgr run started 2015-08-08 00:26:02 -0300 ===========
With arguments: -cells /root/cell_group -patch -rolling
2015-08-08 00:27:07 -0300 :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell …
2015-08-08 00:27:07 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
2015-08-08 00:27:08 -0300 ++++++++++++++++++ Logs so far end ++++++++++
2015-08-08 00:27:08 -0300 :SUCCESS: DONE: Check cells have ssh equivalence for root user.
2015-08-08 00:27:11 -0300 :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute …
2015-08-08 00:27:28 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
.
.
.

I didn’t posted the whole log as this is a big one, but it is here so feel free to take a look at it. And finally we went to database nodes.

  • DB NODES UPGRADE:

[root@exa01db01 5.150701]# ./dbnodeupdate.sh -u -l /u01/install/quarterly_full_jul2015/21339383/Infrastructure/12.1.2.1.2/ExadataDatabaseServer_OL6/p21151982_121212_Linux-x86-64.zip -s

After this procedure thee server get reboot and it takes a while to bring up. If everything is fine with the patching, complete the patching:

[root@exa01db01 5.150701]# ./dbnodeupdate.sh -c

That’s it guys, upgrade finished successfully, everything went smoothly. Hope you enjoyed it!

Storage Exadata: Arquitetura

É isso ai rapaziada! Seguindo com mais um post da série sobre Exadata Database Machine, e pegando uma dica do meu amigo David Siqueira, irei abordar a arquitetura do storage desta máquina. Os servidores que possuem os discos que são apresentados para os bancos que estão no Exadata, possuem atualmente (X5-2) dois tipos a serem escolhidos: Alta Capacidade (High Capacity – HC); ou Extreme Flash (EF). Um servidor com discos de alta capacidade possui doze discos SAS de 10K possuindo 4TB cada um, totalizando 48TB de dado raw, se levarmos em consideração que o diskgroup ficará com a redundância normal, o valor cai para algo em torno de 20TB. O ganho na configuração de extreme flash é quando estamos buscando melhor a escrita no banco, pois pelos dados da Oracle, a escrita pode apresentar o dobro da performance se comparado ao disco de alta capacidade, porém a capacidade de armazenamento do EF cai para 25% em relação ao HC.

Os discos físicos presentes nas controladoras de disco de cada cell node são apresentados como luns ao servidor e baseados nestes que podem ser construídos os cell disks para o software do sistema de storage do Exadata. A visão da controladora de disco é que os discos físicos representam o nível mais baixo de abstração de storage, já para a visão do software do Exadata, os cell disks apresentam o maior nível de abstração de storage dos discos físicos, enquanto que as luns apresentam o menor nível. Seguindo no raciocínio, tendo os cell disks montados, um ou mais grid disks podem ser criados a partir deste, para serem apresentados para a instância do ASM onde enfim serão montados ou adicionados aos diskgroups.

zonebit

 

Os grid disks são montados a partir da faixa (offset) mais “quente” do cell disk, por este motivo que em uma configuração padrão os diskgroups são criados seguindo a ordem de: dados; recuperação; e DBFS. A imagem ao lado ilustra um disco físico, onde as faixas cinzas que estão localizadas na área mais longe do centro do disco, são as que apresentam maior velocidade do disco (hot portion), enquanto que as faixas no tom de salmão possuem menor velocidade (cold portion).

 

Somente a Oracle ACS está autorizada a modificar a estrutura dos cell disks após a entrega da máquina (salvo em casos que exista um acompanhamento via SR no suporte da Oracle). Caso a empresa que adquiriu o Exadata Database Machine altere a estrutura, esta poderá perder o suporte do produto pois a autonomia da empresa para modificar a arquitetura dos discos somente compete a nível dos grid disks em diante. Abaixo segue uma imagem detalhando esta arquitetura desde os discos físicos até os grid disks:

Exadata Storage

O que muda no padrão de adição de grid disks de um ambiente convencional é que, deve-se informar os endereços de IPs (Infiniband 01 e 02) do servidor de storage e o nome dos grid disks que serão apresentados para adicionar ou criar estes nos disk groups, já que na configuração máxima de um Exadata poderá ter até 14 nós de células / servidores de storage (não levando em consideração o Expansion Storage Pack que pode chegar a 19 células). Cada grid disk possui o padrão de nomenclatura como <diskgroup_name>_<cell_disk_type>_<cell_disk_number>_<cell_hostname>. Abaixo estão listados os comandos para listar os discos físicos, lunscell disks e grid disks, e também como adicionar os grid disks aos diskgroups.

physical disk

parted lun

lun

griddisk

SYS@SQL> ALTER DISKGROUP <DISKGROUP_NAME> ADD DISK ‘o/<IP_IB01>;<IP_IB02>/<GRIDDISK_NAME>’;
SYS@SQL> ALTER DISKGROUP DATA ADD DISK ‘o/192.168.10.9;192.168.10.10/DATA_CD_00_exa01cell01’;

Como podemos ver acima, os dois grid disks foram montados utilizando um único cell disk e a partir deste grid disk que conseguimos adicionar ou criar os diskgroups no ASM. Bom galera, espero que tenham gostado e até a próxima. Aquele abraaaaaaaaaaa!!!