Using DBMS_PDB.RECOVER

Hi guys, just passing here to post how you should use DBMS_PDB.RECOVER in order to successfully recover your PDB description XML file if you receive the error ORA-65139: Mismatch between XML metadata file and data file <DATAFILE_PATH> .

To successfully execute this procedure, you must have the description XML file. It’s quite simple when you know exactly what to do. The confusion is that you must point all the PDB’s datafiles excluding the ones from UNDO tablespace. That’s because when you are using a Multitenant Architecture, the UNDO tablespace is used for all pluggable databases. Also, all datafiles should be separated with a comma without any spaces, so it should look like this:

BEGIN
DBMS_PDB.RECOVER(PDB_DESCR_FILE=> ‘/home/oracle/PDB01.xml’, PDB_NAME=>’PDB01′, FILENAMES=> ‘+DATA/PDB01/DATAFILE/sysaux_262_863892639,+DATA/PDB01/DATAFILE/system_261_863892625,
+DATA/PDB01/DATAFILE/users_265_863892763′);
END;
/

After that, you can just execute the statement below, if you don’t want to copy datafiles to a new location:

SQL> CREATE PLUGGABLE DATABASE PDB01 USING ‘/home/oracle/PDB01.xml’ NOCOPY TEMPFILE REUSE;

Quite easy, right?! Yes, but I took some time to use this procedure, so that’s why I’m pointing it right to you. Cya, guys!

CellMetric Tool

Hey guys, it’s been a while since my last post as I’ve been quite busy on the last months at a personal project which could be used to collect some current metrics from a Exadata Storage Server. I got a beta version now, so it’s stills under development but I hope you enjoy this tool. It’s a Java App which was built and tested under ESS image version 12.1.2.1.1.150316.2 and Java version 1.7.0_72 . Java 7.0 is deployed under ESS image version 12.1.2.1.1

Well what this app call CellMetric does is quite simple, it executes CellCLI throw SSH – so Node Equivalency to cellmonitor uses needs to be set correctly -, list the current metrics from the cell that you provided using saving this output to a XML file and then print the results on the screen. That easy. The only attention is that the Database I/O Load Metric information is top ordered listing only the hugest 15 databases. The way how you execute it is java CellMetric -top -cell <cellHostName> . Below you can see an image from the execution time:

CellMetric Image

So just download this CellMetric_v1.2.zip place both file on your Exadata Database Machine and then enjoy it. Cya!!

Patching Exadata QFSP July 2015

Hey, everyone! I’m here with this shortly post about patching Exadata QFSP July 2015. My teammate and I have recently patched our X2-2 Half Rack environment from 11.2.3.3.0.131014.1 to 12.1.2.1.2.150617.1 so I want to THANK them (Vitor Eduardo, Claudio Angerami, Bruno Palma, Anselmo Ribeiro and Edmilson Carmo) for the great job we’ve done. There are no big news, nothing really changed from the other post that I made before, the big key is to pay attention on the ‘Known Issues’ and address them as founded. Also, analyze the RPMs that will be deleted in order to guarantee same functionality as before. After that, if everything is fine your platform should be ready to patch.

So let’s go for it! .Just a point here, we changed the real hostnames and IPs from the servers, cells and switches.

  • IB SWITCHES UPGRADE:

[root@exa01db01 patch_12.1.2.1.2.150617.1]# ./patchmgr -ibswitches /root/ib_switches -upgrade

With arguments: -ibswitches /root/ib_switches -upgrade
2015-08-08 17:01:37 -0300 [INFO] pid file: /var/log/exadatatmp/_EXA_AXE_patchmgr.lock : 98220
PID TTY TIME CMD
2015-08-08 17:01:42 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
2015-08-08 17:01:42 -0300 ++++++++++++++++++ Logs so far end ++++++++++
2015-08-08 17:01:42 -0300 1 of 1 :SUCCESS: DO: Initiate upgrade of InfiniBand switches to 2.1.5-1. Expect up to 15 minutes for each switch
2015-08-08 17:45:23 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
—– InfiniBand switch update process started Sat Aug 8 17:01:42 BRT 2015 —–
[NOTE ] Log file at /var/log/cellos/upgradeIBSwitch.log

[INFO ] List of InfiniBand switches for upgrade: ( exa01sw-ib2 exa01sw-ib3 )
[PROMPT ] Use the default password for all switches? (y/n) [n]:
[PROMPT ] Updating only 2 switch(es). Are you sure you want to continue? (y/n) [n]:
[SUCCESS ] Verifying Network connectivity to exa01sw-ib2
[SUCCESS ] Verifying Network connectivity to exa01sw-ib3
[SUCCESS ] Validating verify-topology output
[INFO ] Proceeding with upgrade of InfiniBand switches to version 2.1.5_1
[INFO ] Master Subnet Manager is set to “exa01sw-ib2” in all Switches

[INFO ] ———- Starting with IBSwitch exa01sw-ib2
[SUCCESS ] Disable Subnet Manager on exa01sw-ib2
[SUCCESS ] Copy firmware packages to exa01sw-ib2
[SUCCESS ] exa01sw-ib2 is at 2.1.3-4. Meets minimal patching level 2.1.3-4
[SUCCESS ] Verifying that /tmp has 120M in exa01sw-ib2, found 138M
[SUCCESS ] Verifying that / has 80M in exa01sw-ib2, found 198M
[SUCCESS ] Verifying that exa01sw-ib2 has 120M free memory, found 239M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth[0,1] for exa01sw-ib2
[SUCCESS ] Verifying that exa01sw-ib2 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-08-08 Time:(HH:MM:SS) 17:03:03
[SUCCESS ] Execute plugin check for Patch Check Prereq on exa01sw-ib2
[SUCCESS ] Pre-update validation on exa01sw-ib2
[INFO ] Starting upgrade on exa01sw-ib2 to 2.1.5_1. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[SUCCESS ] Execute plugin check for Patching on exa01sw-ib2
[SUCCESS ] Load firmware 2.1.5_1 onto exa01sw-ib2
[SUCCESS ] Disable Subnet Manager on exa01sw-ib2
[SUCCESS ] Verify that /conf/configvalid is set to 1 on exa01sw-ib2
[SUCCESS ] Set SMPriority to 5 on exa01sw-ib2
[INFO ] Rebooting exa01sw-ib2. Wait for 240 secs before continuing
[SUCCESS ] Reboot exa01sw-ib2
[SUCCESS ] Restart Subnet Manager on exa01sw-ib2
[INFO ] Starting post-update validation on exa01sw-ib2
[SUCCESS ] Inifiniband switch exa01sw-ib2 is at target patching level
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth[0,1] for exa01sw-ib2
[SUCCESS ] Verifying that exa01sw-ib2 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-08-08 Time:(HH:MM:SS) 17:19:42
[SUCCESS ] Firmware verification on InfiniBand switch exa01sw-ib2
[SUCCESS ] Execute plugin check for Post Patch on exa01sw-ib2
[SUCCESS ] Post-check validation on IBSwitch exa01sw-ib2
[SUCCESS ] Update switch exa01sw-ib2 to 2.1.5_1

[INFO ] ———- Starting with InfiniBand Switch exa01sw-ib3
[SUCCESS ] Disable Subnet Manager on exa01sw-ib3
[SUCCESS ] Copy firmware packages to exa01sw-ib3
[SUCCESS ] exa01sw-ib3 is at 2.1.3-4. Meets minimal patching level 2.1.3-4
[SUCCESS ] Verifying that /tmp has 120M in exa01sw-ib3, found 138M
[SUCCESS ] Verifying that / has 80M in exa01sw-ib3, found 199M
[SUCCESS ] Verifying that exa01sw-ib3 has 120M free memory, found 237M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth[0,1] for exa01sw-ib3
[SUCCESS ] Verifying that exa01sw-ib3 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-08-08 Time:(HH:MM:SS) 17:24:40
[SUCCESS ] Execute plugin check for Patch Check Prereq on exa01sw-ib3
[SUCCESS ] Pre-update validation on exa01sw-ib3
[INFO ] Starting upgrade on exa01sw-ib3 to 2.1.5_1. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[SUCCESS ] Execute plugin check for Patching on exa01sw-ib3
[SUCCESS ] Load firmware 2.1.5_1 onto exa01sw-ib3
[SUCCESS ] Disable Subnet Manager on exa01sw-ib3
[SUCCESS ] Verify that /conf/configvalid is set to 1 on exa01sw-ib3
[SUCCESS ] Set SMPriority to 5 on exa01sw-ib3
[INFO ] Rebooting exa01sw-ib3. Wait for 240 secs before continuing
[SUCCESS ] Reboot exa01sw-ib3
[SUCCESS ] Restart Subnet Manager on exa01sw-ib3
[INFO ] Starting post-update validation on exa01sw-ib3
[SUCCESS ] Inifiniband switch exa01sw-ib3 is at target patching level
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth[0,1] for exa01sw-ib3
[SUCCESS ] Verifying that exa01sw-ib3 has at least 1 NTP Server, found 1
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-08-08 Time:(HH:MM:SS) 17:41:09
[SUCCESS ] Firmware verification on InfiniBand switch exa01sw-ib3
[SUCCESS ] Execute plugin check for Post Patch on exa01sw-ib3
[SUCCESS ] Post-check validation on IBSwitch exa01sw-ib3
[SUCCESS ] Update switch exa01sw-ib3 to 2.1.5_1
[INFO ] InfiniBand Switches ( exa01sw-ib2 exa01sw-ib3 ) updated to 2.1.5_1
[SUCCESS ] Overall status

—– InfiniBand switch update process ended Sat Aug 8 17:45:23 BRT 2015 —–
2015-08-08 17:45:23 -0300 ++++++++++++++++++ Logs so far end ++++++++++
2015-08-08 17:45:23 -0300 1 of 1 :SUCCESS: DONE: Upgrade InfiniBand switch(es) to 2.1.5-1.
================PatchMgr run ended 2015-08-08 17:45:23 -0300 ===========


  • CELL NODES UPGRADE:

For this one, you need to execute -reset_force and -cleanup procedures before upgrade, as below:

[root@ndcing01db01 patch_12.1.2.1.2.150617.1]# ./patchmgr -cells /root/cell_group -reset_force
================PatchMgr run started 2015-08-08 00:21:13 -0300 ===========
With arguments: -cells /root/cell_group -reset_force
2015-08-08 00:21:13 -0300 [INFO] pid file: /var/log/exadatatmp/_EXA_AXE_patchmgr.lock : 72003
PID TTY TIME CMD
[INFO] Reset force was successful.
2015-08-08 00:21:18 -0300 :DONE: reset_force
================PatchMgr run ended 2015-08-08 00:21:18 -0300 ===========

[root@ndcing01db01 patch_12.1.2.1.2.150617.1]# ./patchmgr -cells /root/cell_group -cleanup
================PatchMgr run started 2015-08-08 00:21:45 -0300 ===========
With arguments: -cells /root/cell_group -cleanup
2015-08-08 00:22:50 -0300 :Working: DO: Cleanup …
2015-08-08 00:22:51 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
[INFO] Reset force was successful.
2015-08-08 00:22:51 -0300 ++++++++++++++++++ Logs so far end ++++++++++
2015-08-08 00:22:51 -0300 :SUCCESS: DONE: Cleanup
================PatchMgr run ended 2015-08-08 00:22:51 -0300 ===========

After that, you are ready to patch the cells in a rolling fashion:

[root@ndcing01db01 patch_12.1.2.1.2.150617.1]# ./patchmgr -cells /root/cell_group -patch -rolling
================PatchMgr run started 2015-08-08 00:26:02 -0300 ===========
With arguments: -cells /root/cell_group -patch -rolling
2015-08-08 00:27:07 -0300 :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell …
2015-08-08 00:27:07 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
2015-08-08 00:27:08 -0300 ++++++++++++++++++ Logs so far end ++++++++++
2015-08-08 00:27:08 -0300 :SUCCESS: DONE: Check cells have ssh equivalence for root user.
2015-08-08 00:27:11 -0300 :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute …
2015-08-08 00:27:28 -0300 ++++++++++++++++++ Logs so far begin ++++++++++
.
.
.

I didn’t posted the whole log as this is a big one, but it is here so feel free to take a look at it. And finally we went to database nodes.

  • DB NODES UPGRADE:

[root@exa01db01 5.150701]# ./dbnodeupdate.sh -u -l /u01/install/quarterly_full_jul2015/21339383/Infrastructure/12.1.2.1.2/ExadataDatabaseServer_OL6/p21151982_121212_Linux-x86-64.zip -s

After this procedure thee server get reboot and it takes a while to bring up. If everything is fine with the patching, complete the patching:

[root@exa01db01 5.150701]# ./dbnodeupdate.sh -c

That’s it guys, upgrade finished successfully, everything went smoothly. Hope you enjoyed it!

Error ‘resource ora.driver.afd does not exist’

Hi guys! Recently I’ve being through an issue after a recently PSU has being applied on an environment that has AFD configured and working. The error that is being experienced when you are trying to check the state or configure the AFD which is ASMCMD-9527: AFD is loaded, but resource ora.driver.afd does not exist. So the point is that I didn’t find any official document that address this issue so what I’m doing here is that I’m following the same steps when you applied a PSU on a grid infrastructure that has ACFS drivers loaded (you need first to remove ACFS drivers and after that PSU has being applied, you can load the ACFS drivers back againd). There is a picture bellow that shows the error when you try to configure/deconfigure AFD after you applied the PSU.

AFD_01

So as you can see from the error message, the ACFS driver is the one that is preventing you from configure/deconfigure the AFD on the server, now lets go into the simple steps to fix it. Bring the Oracle Restart down, in order to uninstall the ACFS and AFD drivers, then just re-install them:

AFD_02AFD_05

All the steps were performed using a privileged user (root), after that you can check that the AFD was configured correctly.

AFD_06

So that’s it guys, I hope it helps you! See ya!

Configure AFD on a new server

Hi guys, I’ve seen some fellows complaining about how to configure AFD on a new server. Well, actually Oracle doesn’t mention no way how to do this, but I’ll show you bellow the way to go. There are simple tasks to perform but the most important of them is to apply PSU 1 or later on it, cause AFD may cause your system to crash MOS: 12.1.0.2 Minimum PSU Level to Configure ASM Filter Driver (AFD) (Doc ID 1943048.1). There are two ways of doing this, the first one would be to deploy a Stand-alone Grid Infrastructure as usual and them migrate the disks to AFD and the second one is to perform a software only install and configure it from the begging. On this post, I’ll mention about the second choice which is the one that Oracle doesn’t mention about it.

First thing, perform a software only install on your server. After the installation has finished you need to configure the grid infrastructure as a Stand-alone, so pick the first option and run it as root:

standalone.start

Then start the ora.cssd resource, after that add and start both listener and the ASM instance:

cssd.listener.asm

Now you are good to go to configure the ASM Filter Driver and label the devices which you want to use, but before first you need to stop the Oracle Restart. Also, you need to use root user in order to configure AFD (it gives you an error but you can ignore it):

afd_configure

And for the end, apply the last PSU in order to avoid your system from crashing:

afd_configure

That’s it, now you have a fresh new server configured with AFD. See you!!

Exadata Storage: What is the Secret?

Hello guys!! Today I’ll post about some technologies from Exadata Storage. IMHO, Oracle has highly score when it launch this hardware for it database. The newest version of it is an X5-2 that can has it configuration using flash disks on the storage which it’s called Extreme Flash. Ok, let’s take a look on this machine and the most interesting features that are beside it.

This hardware is made of database servers which host the whole database and clusterware instances. The data is present on the storage nodes (cell nodes) and also the Exadata system software.The communication between this servers uses two infiniband switches which can handle data transfer up to 40 Gbps. Besides that there is a management switch and the PDUs. When we talk about High Availability, this machine is all about it.

OK, but what is the deal about this hardware because if it is all about hardware everyone can “copy and paste” it? It is all about avoiding or reducing I/O that the software can provide. And this can only be achieved because there is a communication from the databases and the storage servers using a protocol called iDB that allows intelligent I/O to be done. When the I/O is requested from the database nodes to the cell nodes, the cell node knows what kind of I/O is occurring and how to deal with it.

Most of the features that will be mentioned ahead are about the Smart Scan concept. This behaviour can only occur when Direct Path is performed on the database, so sequential reads will no have benefit from Smart Scan. Bellow are mentioned some of this features that minimize I/O on the Exadata:

  1. Column Filtering: As the name means, there is a filtering about the columns so when your query that retrieves only one column from a table that has 10 columns, only the selected column is returned to the database server. On a normal environment, the storage would retrieve all the columns and the SGBD would filter it;
  2. Predicate Filtering: Similar to the Column Filtering feature, but this one is happens on the row level. The Exadata Storage can retrieve only the rows that satisfy your query;
  3. Cell Offloading: Normally, the work can be offloaded to the cells. An example would be a query that count all the employees from a company (select count(*) from hr.employees), is work is done on the cell nodes and only the result goes back to the db node. There could be cases that when there is a high workload on the cells, it can’t offload the work and all the rows goes back to the database node as it would in a normal environment;
  4. Storage Indexes: The cell nodes have the ability to analyze the queries so they can build the Storage Indexes (SI). This structure resides on the memory of the cell nodes and they are lost on every restart from the cells. This feature can provide information about the minimum and maximum values from a column, so the Exadata knows exactly what are the blocks to hit. Each table can have a maximum of eight SI;
  5. Join Processing: The Exadata uses the Bloom Filtering technique which is a probabilist method when you join two tables to efficiently test result sets, this can only be used on database using version equal and above to 11.2.0.4;

So the Exadata storage has a technology addressed to the highest data throughput per transaction and this is not recommended for an OLTP environment? No exactly, there are three features that I see as better designed to OLTP environments:

  1. Exadata Smart Flash Cache: This isn’t similar to the Database Smart Flash Cache feature. This feature has a method of write called write-back cache which the data can be first written on the ESFC and than can be write asynchronously on the cell disks presented on the storage nodes. Also, you can choose to com compress the data on the ESFC which gives you a better data usage capacity;
  2. Smart Flash Cache Log: There is a small area which is built on the flash cache from each cell nodes designed to redo logs writes. This is known as Smart Flash Cache Log, so when the cell nodes attempts to write a redo log request it tries to write on both cell disks and flasch cache. The first it gets, it acknowledge the request back to the db node speeding redo logs write which is excellent for OLTP environments;
  3. Join Processing: this feature is a good one for both DW and OLTP environments;

I understand that Smart Flash Cache with write-back enabled can be a good feature for DW environments too, but when we move to high workload environment with high load of data, the data could probably not fit into Smart Flash Cache. Besides that, the Exadata Database Machine has a special feature called HCC (Hybrid Columnar Compression) where the data could be compressed at high levels reducing I/O and enhance the performance for this machine. Well guys, that’s all for now! See you!

Storage Exadata: Architecture

Hello fellows!! Lets go head into a new post about Exadata Database Machine, and having a tip from my friend David Siqueira, I’ll write about the storage archtecture from this machine. On the X5-2, the disks that are address to database that are on the Exadata Database Machine can be choose from two flavors: High Capacity (HC); and Extreme Flash (EF). Both of them has twelve disks, the HC configuration has 10K SAS disks having 4TB size each one, totaling 48TB of raw data but if the diskgroups are built using normal redudancy, this value falls to 20TB of data. The point about EF configuration is all about better write for the data. According to Oracle the EF configuration has the double of speed when writing to disks but the data capacity falls behind 25% if relation to HC.

The physical disks are presented on the disk controller for each cell node which are configured as LUNs on the server and based on this, the Exadata system software build the cell disks from the luns. The physical disks are the lower level of abstraction for the disk controller, while for the Exadata Storage software the luns are the lower level and the cell disks are the higher level. When the cell disks are mounted, one or more grid disks can be build from each cell disk and after that the grid disks can be used to create a diskgroup on an ASM instance.

zonebit

Grid disks are always build using the lowest offset (outer part of the disk ) available from a cell disk until the coldest one, often this are the hottest portion of the disk. That’s why when Oracle setup an Exadata Machine, the diskgroup for data use the lowest offset followed by RECO and DBFS diskgroups. The picture from the left show the gray part of the disk as the hottest and the orange one as the coldest

 

 

Only Oracle ACS is authorized to modify the structure for the cell disks after the Exadata Machine setup (except any open SR at Oracle that allows you). And if the company which acquires the Exadata Database Machine modify this structure, it could lose the support from this product because the company can only customize the grid disks from it. Below there is a picture that shows the relation from physical disk until grid disks:

 

Exadata Storage

When you need to add/create a disk to a diskgroup, you must inform all the infiniband IPs from the storage server and the name of the grids. Each grid disk has a naming standard which is composite of <diskgroup_name>_<cell_disk_type>_<cell_disk_number>_<cell_hostname>. Bellow a listed some commands that are used to list the physical disks, luns, cell disks and grid disks, also there is a command to add a grid disk to a diskgroup:

physical disk

parted lun

lun

griddisk

SYS@SQL> ALTER DISKGROUP <DISKGROUP_NAME> ADD DISK ‘o/<IP_IB01>;<IP_IB02>/<GRIDDISK_NAME>’;
SYS@SQL> ALTER DISKGROUP DATA ADD DISK ‘o/192.168.10.9;192.168.10.10/DATA_CD_00_exa01cell01’;

And as you can see, there were two grid disks built on a single cell disk and then we could add it to the diskgrou on the ASM. Well guys, that’s all for now and I hope you like it. See you next time!