Monday, 30 December 2019

RAC Database Upgrade from 11.2.0.3/4 to 12.1.0.2


Preinstallation

  • Create CRQ for the upgrade
  • Update the Os limits
  • Copy the required files to one node of the RAC
  • Copy PSU  and OPatch to all nodes of the RAC
  • Unzip the database files
  • Create RDBMS response file
  • Deploy software
  • Copy init.ora and tnsnames.ora to new RDBMS Home
  • Upgrade OPatch on each node
  • Create ocm response file
  • Update  config.c with ASM group
  • Apply latest PSU on each node
  • Update Goldengate Library
  • Verify Installation
  • Verify a good backup
  • Pre-Upgrade check

Upgrade the database

  • Turn off cluster database
  • Run Pre-Upgrade SQL
  • Disable Golden Gate  System Trigger
  • Shutdown Database
  • Update /etc/oratab
  • Startup Upgrade
  • Upgrade the database
  • Check the upgrade for errors
  • Run Post-Upgrade SQL
  • Enable GoldenGate  System Trigger
  • Apply PSU  to Database and recompile
  • Reset Cluster and verify
  • Verify correct dump destination
  • Shutdown Database
  • Update OCR and start 
  • Compile system triggers
  • Verify Oratab file

Update DST

Close CRQ and Release  Database

*************************************************************************************************************


Preinstallation


Create CRQ for the Upgrade


Update OS Limits

Calculate the hard and soft limits. These values are a function of memory. The hard limit is total memory  and soft limit is 90% of total memory.

As any user

# Get the memlock hard limit(Hardvalue)
grep  MemTotal: /proc/meminfo | awk '{print $2}'
# Get the memlock soft limit(Hardvalue)
echo $(( `grep  MemTotal: /proc/meminfo | awk '{print $2}'` * 9 / 10 ))

Add the following lines to the /etc/security/limits.conf file

As root user:

#Use the SoftValue and HardValue from above
oracle soft memlock <SoftValue>
oracle hard memlock <HardValue>
grid soft memlock <SoftValue>
grid hard memlock <HardValue>

Copy the required files to one node of the RAC

These instructions assume  the following files are copied to /oraworkspace/software location

RDBMS Software Files
  • p17694377_121020_Linux_x86-64_1of8.zip(rdbms #1)
  • p17694377_121020_Linux_x86-64_2of8.zip(rdbms #2)

Latest PSU

p20299023_121020_Linux-x86-64.zip (PSU 3-copy to all nodes)

Latest OPatch

p6880880_121010_Lunix-x86-64.zip (OPatch -copy to all nodes)


Copy PSU and OPatch to all nodes of the RAC

As Oracle 

cd /oraworkspace/software
scp -p p20299023_121020_Linux-x86-64.zip nodeN:/oraworkspace/software
scp -p p6880880_121010_Lunix-x86-64.zip nodeN:/oraworkspace/software

Unzip the database files
As Oracle:

cd /oraworkspace/software
unzip p17694377_121020_Linux_x86-64_1of8.zip
unzip p17694377_121020_Linux_x86-64_2of8.zip

Create RDBMS response files

As Oracle:

cd /oraworkspace/software
grep -i ^[A-Z] database/response/db_install.rsp

Modify the following variables as per RAC environment you are patching

  • DECLINE_SECURITY_UPDATES=true
  • INVENTORY_LOCATION=/u01/app/oraInventory
  • ORACLE_BASE=/u01/app/oracle
  • ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/PSU3
  • oracle.install.db.BACKUPDBA_GROUP=oinstall
  • oracle.install.db.CLUSTER_NODES=node1,node2
  • oracle.install.db.DBA_GROUP=dba
  • oracle.install.db.DGDBA_GROUP=oinstall
  • oracle.install.db.EEOptionsSelection=true
  • oracle.install.db.InstallEdition=EE
  • oracle.install.db.KMDBA_GROUP=oinstall
  • oracle.install.db.OPER_GROUP=oper
  • oracle.install.option=INSTALL_DB_SWONLY
  • UNIX_GROUP_NAME=oinstall

Deploy Software

As Oracle

cd /oraworkspace/software/database

./runInstaller -silent -responseFile /oraworkspace/software/dbInstall.rsp

Copy init.ora and tnsnames.ora to new RDBMS Home

The following is done on each node

As Oracle 

For each node:

. oraenv
cp - p $ORACLE_HOME/network/admin/tnsnames.ora /u01/app/oracle/product/12.1.0.2/PSU3/network/admin

For Each Database:

.oraenv

cp - p $ORACLE_HOME/dbs/init${ORACLE_SID}.ora ${ORACLE_HOME}/dbs/orapw${ORACLE_SID} \  /u01/app/oracle/product/12.1.0.2/PSU3/dbs
done

Upgrade OPatch on each node

Run the following on each node

As Oracle

cd /u01/app/oracle/product/12.1.0.2/PSU3
unzip -o /oraworkspace/software/p6880880_121010_Linux-x86-65.zip

Create OCM response file:

Check for /var/tmp/ocm.rsp if doesn't exist

As Oracle

The following command has 2 Prompts


1. Email address/User Name: Leave Blank
2. Do you wish to remain uninformed ofsecurity issues(Yes,No)N:Answer 'Y"

export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/PSU3
${ORACLE_HOME}/OPatch/ocm/bin/emocmrsp -no_banner -output /var/tmp/ocm.rsp


Example Output:

${ORACLE_HOME}/OPatch/ocm/bin/emocmrsp -no_banner -output /var/tmp/ocm.rsp
Provide  your email address to be informed of security issues, install and initiate oracle configuration  Manager. Easier for you if you use your My Oracle Support Email address/User Name.

Visit http://www.oracle.com/support/policies.html for details
Email address/Username:

You have not provided an email address for notification of security issues.
Do you Wish to remain  uninformed of security issues([Y]es,[N]o [N]:Y

The ocm configuration  response file  (/var/tmp/ocm.rsp)was successfully created.

Update config.c with ASM group 

Correct the value of ASM group in the  $ORACLE_HOME/rdbms/lib/config.c file on each node

This will result in ASM  group  set after the PSU  relink in the following steps

As Oracle

export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/PSU3
sed -t 's/SS_ASM_GRP ""/SS_ASM_GRP "asmadmin"/' ${ORACLE_HOME}/rdbms/lib/config.c

Apply latest PSU on each Node

As Oracle

. If the grid patch will be used 

cd /oraworkspace/software
export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/PSU3
${ORACLE_HOME}/OPatch/opatch apply /orworkspace/software/gpsu/20485724/20299023 -oh ${ORACLE_HOME} -ocmrf /var/tmp/ocm.rsp -silent

. Ifthe RDBMS patch will be used


cd /oraworkspace/software
mkdir psu
cd psu
unzip ../p20299023_121020_Linux-x86-64.zip
export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/PSU3

${ORACLE_HOME}/OPatch/opatch apply /orworkspace/software/psu/20299023 -oh ${ORACLE_HOME} -ocmrf /var/tmp/ocm.rsp -silent

Verify the output mentions the PSU to all RAC nodes

Example:

All nodes have been patched. You may start oracle instances on the local system and nodes

Composite patch 20299023 successfully applied .
OPatch session completed with warnings.
log file location: /u01/app/oracle/product/12.1.0.2/PSU3/cfgtoollogs/opatch/opaatch2019-12-30_18-56-12PM_1.log
OPatch completed with warnings.

Update Goldengate Library

As Oracle

. oraenv
ln -s ${ORACLE_HOME}/lib/libnnz12.so ${ORACLE_HOME}/lib/libnnz11.so

Verify Installation

As Oracle:

Verify software installed on all nodes

for node in ${nodes} ; do 
ssh -q ${node} du -sk /u01/app/oracle/product/12.1.0.2/PSU3
done


Should looks something like the  following . Verify the directory   size is similar.
5863920 /u01/app/oracle/product/12.1.0.2/PSU3
6260832 /u01/app/oracle/product/12.1.0.2/PSU3

Check the inventory to insure all appears to be in order

for node in ${nodes} ; do 
ssh -q ${node} tail /u01/app/oraInventory/ContentsXML/inventory.xml
done

Verify PSU applied on all nodes 

for node in ${nodes} ; do 
ssh -q  ${node} /u01/app/oracle/product/12.1.0.2/PSU3/OPatch/opatch_lsinventory
done

Verify a good backup

Before any change is made to the database , Insure a good backup is in place.

. Check the crontab for the regularly scheduled backup
. Check the logfile to insure the last backup was successful.
. Check each node if RAC if not found on node1 . The backup only needs to run from one node of the RAC.

. If backup is not scheduled or the backups are not successful

Run export backup and create guarantee restore point.


Pre-Upgrade Check

Run the pre-upgrade check on the database to upgrade . Will need to be  run individually for each database to upgrade.

As Oracle for each database

# Run for each instance to be upgraded.

. oraenv
sqlplus /nolog <<EndSQL
connect / as sysdba
@/u01/app/oracle/product/12.1.0.2/PSU3/rdbms/admin/preupgrd.sql
exit
EndSQL


Review the output from the above command for each database . Will need  to run the required items either before or after the upgrade

./u01/app/oracle/cfgtoollogs/<DBName>/preupgrade/preupgrade.log
./u01/app/oracle/cfgtoollogs/<DBName>/preupgrade/preupgrade_fixups.sql
./u01/app/oracle/cfgtoollogs/<DBName>/preupgrade/postupgrade_fixups.sql

Upgrade the database


Turn Off Cluster Database

As Oracle

. oraenv

sqlplus /nolog <<EndSQL
connect / as sysdba
alter system set cluster_database=false scope=spfile sid='*';
EndSQL


Run Pre-Upgrade SQL

Run the Pre-Upgrade script and the manual items from preupgrd.sql executed above

As Oracle

. oraenv

sqlplus /nolog <<EndSQL
connect / as sysdba
purge dba_recyclebin;
column Name new_value DBName
select name from v\$database;
@/u01/app/oracle/cfgtoollogs/&DBName./preupgrade/preupgrade_fixups.sql
EndSQL

Note that the Preupgrd_fixups.sql script doesn't execute all required items. The "MANUAL ACTION REQUIRED" manually.

Disable Golden Gate System Trigger

sqlplus /nolog <<EndSQL
connect / as sysdba
alter trigger sys.GGS_DDL_TRIGGER_BEFORE disable;
EndSQL

The above will fail if golden gate is not installed. Any error can be ignored.


Shutdown Database

As Oracle

. oraenv
srvctl stop database -d <DBNAME>

Update /etc/oratab

As Oracle, Update /etc/oratab file to the new ORACLE_HOME on all nodes 

Startup Upgrade

Startup the database using new ORACLE_HOME


As Oracle

. oraenv

sqlplus /nolog <<EndSQL
connect / as sydba
Startup Upgrade
EndSQL

Upgrade the Database

As Oracle

. oraenv

cd $ORACLE_HOME/rdbms/admin
$ORACLE_HOME/perl/bin/perl/ catctl.pl -n 8 -l ${ORACLE_HOME}/rdbms/log catupgrd.sql


Example Output

$ORACLE_HOME/perl/bin/perl/ catctl.pl -n 8 -l ${ORACLE_HOME}/rdbms/log catupgrd.sql
...
catctl.pl version 12.1.0.2
Oracle Base       =/u01/app/oracle
Analyzing file catupgrd.sql
Log files in  /u01/app/oracle/product/12.1.0.2/PSU3/rdbms/admin
catcon: ALL catcon-related output will be written to catupgrd_catcon_30168.lst
catcon: See catupgrd*.log files for output generated by script
catcon: See capupgrd_*.lst files for spool files, if any 
Number of Cpus    =4
SQL Process Count =8
--------------------------------------------------------------
Phases [0-73]
Serial  Phase #: 0 Files: 1 Time: 103s
Serial  Phase #: 1 Files: 5 Time: 44s
.....................................
.....................................
.....................................
Serial  Phase #: 72 Files: 1 Time: 0s
Serial  Phase #: 73 Files: 5 Time: 23

Grand Total Time: 952s
LOG FILES: (Catupgrd*.log)
Upgrade Summary  Report Located in :
/u01/app/oracle/product/12.1.0.2/PSU3/cfgtoollogs/DBNAME/upgrade/upg_summary.log
Grand Total Upgrade Time: [0d:0h:15m:52s]

Check Upgrade for errors

As Oracle 

cd ${ORACLE_HOME/rdbms/log
grep ^ORA- catupgrd*

Run Post Upgrade SQL

Run the post-upgrade script and manual items from the preupgrd.sql executed above 

As Oracle

. oraenv
sqlplus /nolog <<EndSQL
connect / as sysdba
purge dba_recyclebin;
column Name new_value DBName
select name from v\$database;
@/u01/app/oracle/cfgtoollogs/&DBName./preupgrade/postupgrade_fixups.sql
EndSQL

Enable Goldengate System Trigger


sqlplus /nolog <<EndSQL
connect / as sysdba
alter trigger sys.GGS_DDL_TRIGGER_BEFORE enable;
EndSQL

The above will fail if golden gate is not installed. Any error can be ignored.

Apply PSU to Database and Recompile

Check for invalid objects and run the final upgrade verification script

As oracle
. oraenv
sqlplus /nolog <<EndSQL
connect / as sysdba
startup
EndSQL
${ORACLE_HOME}/OPatch/datapatch -apply 20299023

sqlplus /nolog <<EndSQL
connect / as sysdba
@?/rdbms/admin/utlrp.sql
@?/rdbms/admin/utluiobj.sql
@?/rdbms/admin/utlul21s.sql
EndSQL

Reset Cluster and Verify

As Oracle 

. oraenv
sqlplus /nolog <<EndSQL

connect / as sysdba

alter system set cluster_database=true scope=spfile and sid='*';
set linesize 100
column banner format a80
column version format a10
column action format a10
column status format a10
column description format a50
select * from v\$version;
select patch_id,version,action,status,description from dba_registry_sqlpatch order by action_time;
EndSQL


Verify Correct Dump Destinations

Be Sure Diagnostic_dest is properly set and *dump_dest  default

As Oracle

. oraenv
sqlplus /nolog <<EndSQL
connect / as sysdba
show parameter diagnostic_dest

alter  system set diagnostic_dest='/oratrace' scope=spfile sid='*';

show parameter background_dump_dest


alter  system reset background_dump_dest scope=spfile sid='*';


show parameter core_dump_dest



alter  system reset core_dump_dest scope=spfile sid='*';

show parameter user_dump_dest




alter  system reset user_dump_dest scope=spfile sid='*';

EndSQL

Anyone of the above may result in a error that can be ignored.

.ORA-32010: Cannot find entry to delete in SPFILE



Set correct audit_file_dest and insure directories exist




sqlplus /nolog <<EndSQL

connect / as sysdba

column C1 new_name C!

select  'set audit_file_dest=''/oradbaudit/admin//'||name||'/adump''scope=spfilesid=''*''' C1 from v\$database;


alter system &C1.
/
set pagesize 0
set linesize 120
select 'ssh -q ' ||host_name||" mkdir -p /oradbaudit/admin/' ||name||'/adump from v\$database,gv\instance;
EndSQL


Run the above 'ssh -q..." statements from the command line

Shutdown Database
As Oracle
. oraenv


sqlplus /nolog <<EndSQL
connect / as sysdba
shut immediate;
EndSQL


Update OCR and start

As Oracle

. oraenv

srvctl upgrade database -d <DBNAME> -o /u01/app/oracle/product/12.1.0.2/PSU3
srvctl start database d <DBName>

Compile System Triggers

As Oracle
. oraenv


sqlplus /nolog <<EndSQL
connect / as sysdba
select 'alter trigger '||owner||','||object_name||' compile;' from dba_objects where status='INVALID' and object_type='TRIGGER';
EndSQL

Run the above "alter trigger...."statements from the command line

Verify Oratab file

As Oracle

verify /etc/oratab properly updated on all nodes

Update DST

Update DST to version 18 Follow below link to instructions.