Tuesday, 7 January 2020

HA GRID Upgrade From 11.2.0.3/4 To 12.1.0.2


Upgrade HAS grid from Oracle version 11.2.0.3/4 to 12.1.0.2.

Preinstallation
  • Create CRQ
  • Create Grid Directory for 12.1.0.2
  • Clean the Software Directory
  • Copy the required files
  • Unzip the files
  • Verify System Parameters
  • Pre-Requisite Check
  • Create grid response file
  • Deploy software
  • Run root.sh and save upgrade command syntax
  • Save HAS OCR Cinfiguration
  • Execute rootupgrade.sh
  • Verify Installation
  • Complete preinstallation items


Upgrade HAS Grid
  • Stop Instances
  • Stop Listener and copy listener.ora,sqlnet.ora and ASM Password file
  • Deconfigure old home
  • Update  /etc/oratab
  • Configure New Home
  • Verify evmd running
  • Configure Listener and set cssd to auto start
  • Verify HAS startup
  • Add ASM to OCR
  • Add Database to OCR
  • Start all Databases
  • Verify all databases are up and running
  • Final check of cluster version and status

  Apply The latest PSU
  • Update OPatch
  • Unzip Patch
  • Create OCM response file
  • Shutdown Databases
  • Apply grid PSU
  • Startup Databases
  • Verify all databases are up and running
  • Remove TFA
  • Update ASM and Listener home in OEM
  • Close CRQ 

 *************************************************************************************************************
Overview

 Upgrade HAS grid from Oracle version 11.2.0.3/4 to 12.1.0.2. 

 Preinstallation

Create CRQ

  A CRQ is required for an upgrade of grid

 Create Grid Directory  for 12.1.0.2

 As root

 mkdir -p /u01/app/12.1.0.2/grid
chown -R grid:oinstall /u01/app/12.1.0.2 /u01/app/12.1.0.2/grid
chmode -R 755 /u01/app/12.1.0.2 /u01/app/12.1.0.2/grid

 As oracle

 cd /oraworkspace/software
mkdir -p oracle12cR1
chmod -R 777 oracle12cR1
mkdir -p opatch psu_201610 psu
mkdir -p grid_psu_201610 db db_psu_201610
Clean the software directory

 Remove all previous installation software from /oraworkspace/software and change the directory group so our personal ID's can copy software directory (dba group)

 As root

 cd /oraworkspace/software
rm -rf database grid psu gpsu
chgrp dba /oraworkspace/software
chmod 775 /oraworkspace/software
The above is a minimum , Consider clearing all files from /oraworkspace/software

Copy required files

 Please use latest version

 Grid software files

 p17694377_12120_Linux-x86-64_3of8.zip(grid#1)
p17694377_12120_Linux-x86-64_4of8.zip(grid#2)

 PSU Patch software

 p20485724_121020_Linux-x86-64.zip(PSU 3-Copy to all nodes)
OPatch software 

 p6880880_121010_Linux-x86-64.zip(OPatch-copy to all nodes)

 Unzip the files

 As grid

 cd /oraworkspace/software
unzip p17694377_12120_Linux-x86-64_3of8.zip
unzip p17694377_12120_Linux-x86-64_4of8.zip

 Verify system Parameters
  • As root : Add kernel_panic_on_oops=1 to /etc/sysctl.conf file
  • Must have at least 4g memory to complete the upgrade. grep MemTotal /proc/meminfo
  • Result larger than 4GB(4195205.0 KB). Request n increase in size of VM if value is not large enough


 Pre-Requisite Check 

 To Upgrade from previous version to 12.1.0.2 make sure all the pre-requisite checks are met. Pre-Requisited check may fail for some patches. The prerequisite check uses following syntax.

 As grid 

 cd /oraworkspace/software/grid
runcluvfy.sh stage -pre hacfg [-osdba <osdba_group>][orainv <orainventory_group>][-fixup][-fixupnoexec][-methode sudo -user <user_name>[-location <dir_path>]|-method root][-verbose] 
Example (as grid)

 unset ORACLE_BASE
unset ORACLE_HOME
unset ORACLE_SID
cd /oraworkspace/software/grid
./runluvfy.sh stage -pre hacfg -fixup

Example Output:

 ./runluvfy.sh stage -pre hacfg -fixup
Performing pre-checks for Oracle restart configuration
Total memory check passed
Available memory check passed
Swap space check passed
....
Check passed on nodes

 Following is the list of fixable prerequisites selected to fix in this session
***********************************************************
-------------------   ---------------------- -----------------------
Check Failed.         Failed on nodes        Reboot Required?
-------------------   ---------------------- -----------------------
OS Kernel Parameter:  oradb1                 NO
panic_on_oops

 Execute "/tmp/CVU_12.1.0.2.0_grid/runfixup.sh" as root user on nodes "oradb1" to perform the fix up operations manually.
Press ENTER key to continue after execution of "/tmp/CVU_12.1.0.2.0_grid/runfixup.sh" has completed on nodes "oradb1"

 FIX: OS kernel parameter :panic_on_oops

 "OS Kernel parameter : panic_on_oops" was successfully fixed on all the applicable nodes

 Follow the instrctions : Open a new window , login as root and run /tmp/CVU_12.1.0.2.0_grid_runfixup.sh , then continue

 Create grid response file

 As grid

 cd /oraworkspace/software
grep -i ^[A-Z] grid/response/grid_install.rsp |sort>grid.rsp

 Modify the following variables as per environment you are patching

 .INVENTORY_LOCATION=/u01/app/oraInventory
.ORACLE_BASE=/u01/app/grid
.ORACLE_HOME=/u01/app/12.1.0.2/grid
.oracle.install.asm.diskGroup.redundancy=
.oracle.install.asm.OSASM=asmadmin
.oracle.install.asm.OSDBA=asmdba
.oracle.install.asm.OSOPER=asmoper
.oracle.install.crs.config.autoConfigureClusterNodeVIP=false
.oracle.installer.autoupdates.option=SKIP_UPDATES

Deploy software

 As grid
cd /oraworkspace/software/grid
./runInstaller -silent -responseFile oraworkspace/software/grid.rsp 

 Example Output:

 cd /oraworkspace/software/grid
./runInstaller -silent -responseFile oraworkspace/software/grid.rsp 
Starting Oracle Universal Installer...
Checking Temp space:must be greater than 415MB. Actual 1655 MB Passed
Checking swap space:must be greater than 150MG. Actual 31923 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2019-12-31_14-23-34PM , Pleas wait ...[WARNING][INS-13014]
Cause:Some of the optional prerequisites are not met .See the logs for details , /u01/app/oraInventory/logs/InstallActions201-12-31-14_23_34.log
Action:Identify the list of failed prerequisite check from the log /u01/app/oraInventory/logs/InstallActions201-12-31-14_23_34.log

 You can find the log of this install session at: 
/u01/app/oraInventory/logs/InstallActions201-12-31-14_23_34.log
As root user, execute the following script(s):
  1. /u01/app/12.1.0.2/grid/root.sh

 Successfully setup software.
As install user, execute the following script to complete the configuration.
  1. /u01/app/12.1.0.2/grid/cfgToolAllCommands RESPONSE_FILE=<response_file>.
Note:

 1. This script must be run on the same host from where installer was run.
2. This script needs a small password properties file for configuration assistance that require password.

 Run root.sh and save upgrade command syntax

 As root

 /u01/app/12.1.0.2/grid/root.sh
Check and save the log file to extract the required command for update that is used later
Save HAS OCR Configuration

 The change in the grid ORACLE_HOME causes the loss of the HAS equivalent of the OCR
This information is saved for later use to reconstruct after the new home is used.

 As grid

 srvctl config listener > /var/tmp/grid.info 
srvctl config asm >> /var/tmp/grid.info
for db in `srvctl config database`;do
 srvctl config database -d ${DB}>>/var/tmp/grid.info
done
Execute rootupgrade.sh

 As root , Upgrade the software

 /u01/app/12.1.0.2/grid/rootupgrade.sh
Sample output:

 Check /u01/app/12.1.0.2/grid/install/root_oradb1_2019-12-31_21-30-28.log for the output of root script

 Verify successful completion of the script by examining the log file and checking versions.

 Verify Installation

 As grid
  • Verify software installed

     du -sk /u01/app/12.1.0.2/grid
    Should look something like the following where the directory        size is similar

     6843672 /u01/app/12.1.0.2/grid
  • Verify software version,release and active versions
    crsctl query has releaseversion
    crsctl query has softwareversion

    Expect output similar to


    Oracle High Availability Services release version on the local      node is [11.2.0.4.0]
    Oracle High Availability Services version on the local node is      [11.2.0.4.0]

The following steps requires 2 database outages(one for 12.1.0.2 and the other for PSU)

Upgrade HAS Grid

Stop Instances

As grid,Stop all instances if $ORACLE_HOME is not the same versionas grid. will need to stop as the oracle used

for db in`srvctl config database`;do
 echo "stop database ${db}"
 srvctl stop database -d ${db}
done

Stop listener and copy listener.ora,sqlnet.ora and ASM password file

As grid

srvctl stop listener
cp /u01/app/11.2.0.4/grid/network/admin/listener.ora /u01/app/12.1.0.2/grid/network/admin
cp /u01/app/11.2.0.4/grid/network/admin/sqlnet.ora /u01/app/12.1.0.2/grid/network/admin

cp /u01/app/11.2.0.4/grid/dbs/orapw+ASM /u01/app/12.1.0.2/grid/dbs

Deconfigure old home

As root

ORACLE_SID=+ASM
ORAENV_ASK="NO"
.oraenv -s
unset ORAENV_ASK
${ORACLE_HOME}/crs/install/roothas.pl -deconfig -force

Example output:

#${ORACLE_HOME}/cr/install/roothas -pl -deconfig -force
Using configuration parameter file :/u01/app/12.1.0.2/grid/crs/install/crsconfig_params
CRS resource for listeners are still configured
PRKO-2573: ONS daemon is already stopped
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'oradb1'
CRS-2673: Attempting to stop 'ora.FRADG.dg' on 'oradb1'
CRS-2677: Stop of 'ora.FRADG.dg' on 'oradb1' succeeded 
CRS-2677: Stop of 'ora.DATADG.dg' on 'oradb1' succeeded 
CRS-2673: Attempting to stop 'ora.asm' on 'oradb1'
CRS-2677: Stop of 'ora.asm' on 'oradb1' succeeded 
CRS-2673: Attempting to stop 'ora.cssd' on 'oradb1'
CRS-2677: Stop of 'ora.cssd' on 'oradb1' succeeded 
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oradb1'
CRS-2673: Attempting to stop 'ora.evmd' on 'oradb1'
CRS-2677: Stop of 'ora.evmd' on 'oradb1' succeeded 
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oradb1' has completed.
CRS-4133: Oracle High Availability Services has been stopped.

Successfully deconfigured Oracle Restart Stack

Update /etc/oratab

Update the ASM entry in the /etc/oratab
+ASM:/u01/app/12.1.0.2/grid:N


Configure New Home

As root

ORACLE_SID=+ASM
ORAENV_ASK="NO"
.oraenv -s
unset ORAENV_ASK
/u01/app/12.1.0.2/grid/perl/bin/perl -I /u01/app/12.1.0.2/grid/perl/lib -I /u01/app/12.1.0.2/grid/crs/install /u01/app/12.1.0.2/grid/crs/install/roothas.pl

Example output:

/u01/app/12.1.0.2/grid/perl/bin/perl -I /u01/app/12.1.0.2/grid/perl/lib -I /u01/app/12.1.0.2/grid/crs/install /u01/app/12.1.0.2/grid/crs/install/roothas.pl

Using configuration parameter file :/u01/app/12.1.0.2/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'grid',privgrp 'oinstall'...
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary  OCR keys.
Creating OCR key for user 'root',privgrp 'root'...
Operation successful.
CRS-4664:Node oradb1 successfully pinned
2019/12/31 21:38:54 CLSRSC-330:Adding Clusterware entries to file 'oracle-ohasd.conf'

oradb1   2019/12/31 21:39:14   /u01/app/12.1.0.2/grid/cdata/oradb1/backup_20191231_21:39:14.olr

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oradb1'
CRS-2673: Attempting to stop 'ora.evmd' on 'oradb1'
CRS-2677: Stop of 'ora.evmd' on 'oradb1' succeeded 
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oradb1' has completed.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.

2019/12/31 21:41:23 CLSRSC:327: Successfully configured Oracle Restart for a standalone server

Verify evmd running

As grid

crsctl status resource -t


Example output
--------------------------------------------------------------------
Name          Target   State      Server      State Details
--------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------
ora.ins       OFFLINE  OFFLINE    oradb1      STABLE
--------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------
ora.cssd
    1         OFFLINE  OFFLINE                STABLE
ora.diskmon
    1         OFFLINE  OFFLINE                STABLE
ora.evmd
    1         OFFLINE  OFFLINE    oradb1      STABLE
--------------------------------------------------------------------


Configure listener and set cssd to auto start

 As grid

 crsctl modify resource ora.cssd -init -attr "AUTO_START=always"
/u01/app/12.1.0.2/grid/bin/srvctl add listener -l LISTENER -o /u01/app/12.1.0.2/grid -p "TCP:1521"
/u01/app/12.1.0.2/grid/bin/srvctl start listener -l LISTENER

Verify HAS startup

 As root


ORACLE_SID=+ASM
ORAENV_ASK="NO"
.oraenv -s
unset ORAENV_ASK
crsctl stop has
crsctl start has


 Example output
#crsctl stop has


CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oradb1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oradb1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oradb1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'oradb1'
CRS-2677: Stop of 'ora.evmd' on 'oradb1' succeeded 
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oradb1' has completed.
CRS-4133: Oracle High Availability Services has been stopped.
#crsctl start has
CRS-4123: Oracle High Availability Services has been started.

 As grid

 crsctl status resource -t

Example output
--------------------------------------------------------------------
Name               Target   State      Server      State Details
--------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------
ora.LISTENER.lsnr  ONLINE  ONLINE      oradb1      STABLE
ora.ons            OFFLINE OFFLINE     oradb1      STABLE
--------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------
ora.cssd
    1              ONLINE  ONLINE      oradb1      STABLE
ora.diskmon
    1              OFFLINE OFFLINE                 STABLE
ora.evmd
    1              ONLINE  ONLINE      oradb1      STABLE
--------------------------------------------------------------------


Add ASM to OCR

 As grid

 Note that the asmSPFileName is from the srvctl config as command above

 Example only:

--------------------------------------------------------------------
$/u01/app/12.1.0.2/grid/bin/srvctl add asm -l LISTENER -p "asmSPFileName -d "/dev/ASM"
$/u01/app/12.1.0.2/grid/bin/srvctl start asm
$crsctl status resource -t

Example output
--------------------------------------------------------------------
Name               Target   State      Server      State Details
--------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------
ora.DATADG.dg      ONLINE   ONLINE      oradb1      STABLE
ora.FRADG.dg       ONLINE   ONLINE      oradb1      STABLE
ora.LISTENER.lsnr  ONLINE   ONLINE      oradb1      STABLE
ora.asm            ONLINE   ONLINE      oradb1      Started,STABLE
ora.ons            OFFLINE  OFFLINE     oradb1      STABLE
--------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------
ora.cssd
    1              ONLINE   ONLINE      oradb1      STABLE
ora.diskmon
    1              OFFLINE  OFFLINE                 STABLE
ora.evmd
    1              ONLINE   ONLINE      oradb1      STABLE
--------------------------------------------------------------------

Add Databases to OCR

 As oracle
Note that the parameters in the example command is from the "srvctl config database" commands above


 Example Code:

 #/u01/app/oracle/product/11.2.0.4/dbhome_1/bin/srvctl add database -d abc -o /u01/app/oracle/product/11.2.0.4/dbhome_1 -m sjl.ic3.com -p "+DATADG/abc/spfileabc.ora" -r PRIMARY -s open -t immediate -n abc -i abc -y AUTOMATIC -a DATADG,FRADG

Add all databases using a command similar to above


Start all databases


 As oracle

for db in `srvctl config database` ;do
 echo "Start database ${db}"
 srvctl start database -d 4{db}
done


Verify all database are up and running

 As oracle

 ps -ef|grep pmon
.oraenv DBName


For each database above, set the environments and verify using the following

sqlplus /nolog >>Endsql
connect / as sysdba
select count(1) from dba_source;
exit
EndSQL

As grid

 crsctl staus resource -t


Example output:
--------------------------------------------------------------------
Name               Target   State      Server      State Details
--------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------
ora.DATADG.dg      ONLINE   ONLINE      oradb1      STABLE
ora.FRADG.dg       ONLINE   ONLINE      oradb1      STABLE
ora.LISTENER.lsnr  ONLINE   ONLINE      oradb1      STABLE
ora.asm            ONLINE   ONLINE      oradb1      Started,STABLE
ora.ons            OFFLINE  OFFLINE     oradb1      STABLE
--------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------
ora.cssd
    1              ONLINE   ONLINE      oradb1      STABLE
ora.diskmon
    1              OFFLINE  OFFLINE                 STABLE
ora.evmd
    1              ONLINE   ONLINE      oradb1      STABLE
ora.ABC.db
    1              ONLINE   ONLINE      oradb1      Open,STABLE
--------------------------------------------------------------------

Final check of cluster version and status

 As grid

 crsctl query has releaseversion
crsctl query has softwareversion
crsctl check has

 Example Output:
#crsctl query has releaseversion
Oracle High Availability Services release version on the local      node is [12.1.0.2.0]
#crsctl query has softwareversion
Oracle High Availability Services version on the local      node is [12.1.0.2.0]
#crsctl check has
CRS-4638: Oracle High Availability Services is online

 Apply the latest PSU

 Apply the latest PSU

 Update OPatch

 As grid

 cd ${ORACLE_HOME}
unzip -o /oraworkspace/software/p6880880_121010_Linux-x86-64.zip
Unzip patch

 As grid

 cd /oraworkspace/software
mkdir gpsu
cd gpsu
unzip -o ../p20485724_121020_Linux-x86-64.zip

 Create OCM response file

 As grid

The following command has 2-prompts

 1. Email address/User Name: Leave Blank
2. Do yo wish to remain uninformed of security issues(Yes,No)N: Answer "Y"

  ${ORACLE_HOME}/OPatch/ocm/bin/emocmrsp -no_banner -output /var/tmp/ocm.rsp
Example output:

 ${ORACLE_HOME}/OPatch/ocm/bin/emocmrsp -no_banner -output /var/tmp/ocm.rsp
Provide your email address to be informed of security issues,install and initiate Oracle Configuration Manager. Easier for you if you use your My Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided as email address for notification of security issues.Do you wish to remain uninformed of security issues([Y]es,[N]o)[N]:Y

 The OCM configuration response file (/var/tmp/ocm.rsp) was successfully created.
Shutdown Databases:

As oracle or grid

   for db in `srvctl config database `;do
   echo "stop database ${db}"
   srvctl stop database -d ${db}
  done

 Apply grid PSU

 As root, Set Environment as +ASM and apply PSU

 ORACLE_SID=+ASM
ORAENV_ASK="NO"
.oranev
echo $ORACLE_SID
echo $ORACLE_HOME
${ORACLE_HOME}/OPatch/opatchauto apply /oraworkspace/software/20485724 -oh ${ORACLE_HOME} -ocmrf /var/tmp/ocm.rsp
Example output


OPatch Automation Tool
Copyright (C) 2014, Oracle Corporation . All rights reserved.

OPatchauto Version  :12.1.0.1.7
OUI Version         :12.1.0.2.0
Running from        :/u01/app/12.1.0.2/grid      
opatchauto log file :/u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/20485724/opatch_gi_2019-12-31_20-31-12_deploy.log

 Parameter validation:Successful

User specified following Grid Infrastructure home:
/u01/app/12.1.0.2/grid 



Configuration Validation : Successful

Patch Location:/oraworkspace/software/gpsu/20485724
Grid Infrastructure Patch(es):20299023 19392590 19392604
RAC Patch(es): 20299023 19392604


Patch Validation:Successful  

Applying Patch(es) on "/u01/app/12.1.0.2/grid"...
Patch "/oraworkspace/software/gpsu/20485724/20299023" successfully applied to "/u01/app/12.1.0.2/grid".
Patch "/oraworkspace/software/gpsu/20485724/19392590" successfully applied to "/u01/app/12.1.0.2/grid".
Patch "/oraworkspace/software/gpsu/20485724/19392604" successfully applied to "/u01/app/12.1.0.2/grid".

Starting CRS.....Successful

Apply Summary:
Following patch(es) successfully installed :
GI Home: /u01/app/12.1.0.2/grid:20299023,19392590,19392604
opatchauto succeeded 



Startup Databases

 As oracle or grid

   for db in `srvctl config database `;do
   echo "start database ${db}"
   srvctl start database -d ${db}
  done


Verify all database are up and running

As oracle

ps -ef|grep pmon
.oraenv DBName


For each database above, set the environments and verify using the following

sqlplus /nolog >>Endsql
connect / as sysdba
select count(1) from dba_source;
exit
EndSQL

As grid

 crsctl staus resource -t


Example output:
--------------------------------------------------------------------
Name               Target   State      Server      State Details
--------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------
ora.DATADG.dg      ONLINE   ONLINE      oradb1      STABLE
ora.FRADG.dg       ONLINE   ONLINE      oradb1      STABLE
ora.LISTENER.lsnr  ONLINE   ONLINE      oradb1      STABLE
ora.asm            ONLINE   ONLINE      oradb1      Started,STABLE
ora.ons            OFFLINE  OFFLINE     oradb1      STABLE
--------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------
ora.cssd
    1              ONLINE   ONLINE      oradb1      STABLE
ora.diskmon
    1              OFFLINE  OFFLINE                 STABLE
ora.evmd
    1              ONLINE   ONLINE      oradb1      STABLE
ora.ABC.db
    1              ONLINE   ONLINE      oradb1      Open,STABLE
--------------------------------------------------------------------

Remove TFA 

 Follow below link to Remove TFA

 Update ASM and Listener home in OEM
The listener and ASM configuration must be updated with new ORACLE_HOME
. Update Listener
  .Open OEM and enter the  hostname in "Search target name"box in the upper right
  .Select the "Target Name" corresponding with the "Listener" "Target Type"
  .From the "Oracle Listener" drop down (Upper Left) select "Target Setup"/"Monitoring configuration"
  .Update the directories paths for the  "Listener.ora Directory " and Oracle Home Fields.

 . Update ASM

   .Select the "Target Name" corresponding with the "Automatic Storage Management"  "Target Type"
  . From the "Automatic storage Management" drop down list (Upper left) select "Target Setup"/"Monitoring configuration"
  . Update the value of the "ORACLE HOME PATH". Test connection to ensure all is in order.
  . Click OK to save

 Close CRQ
Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)