How to Patch Oracle Restart 19c and Oracle Database Using Out-Of-Place Switch Home

Let me show you how I patch Oracle Restart and Oracle Database 19c using the out-of-place method by switching to the new Oracle homes.

The advantages of this solution:

I get more control over the process
I can perform the entire operation with just one database restart
I can create my Oracle homes using gold images
I can prepare the new Oracle homes in advance
Overall, I find this method less riskier

My demo system

Single instance database in Oracle Restart configuration
Runs Oracle Linux
GI and database home are currently on 19.24

I want to:

patch to 19.25
patch both the GI and database home in one operation

Preparation

I need to download:

The base releases of:
- Oracle Grid Infrastructure (LINUX.X64_193000_grid_home.zip)
- Oracle Database (LINUX.X64_193000_db_home.zip)
Latest OPatch from My Oracle Support (6880880).
Patches from My Oracle Support:
- 19.25 Release Update for Grid Infrastructure (36916690)
- Matching OJVM bundle patch (36878697)
- Matching Data Pump bundle patch (36682332)

You can use AutoUpgrade to easily download GI patches.

I place the software in /u01/software.

How to Patch Oracle Restart 19c and Oracle Database

1. Prepare a New GI Home

I can do this in advance. It doesn’t affect my current environment and doesn’t cause any downtime.

I need to create a folder for the new GI home. I must do this as root:

[root@node1]$ mkdir -p /u01/app/19.25.0/grid
[root@node1]$ chown -R grid:oinstall /u01/app/19.25.0
[root@node1]$ chmod -R 775 /u01/app/19.25.0

I switch to the GI home owner, grid.

I extract the base release of Oracle Grid Infrastructure into the new GI home:

[grid@node1]$ export OLDGRIDHOME=$ORACLE_HOME
[grid@node1]$ export NEWGRIDHOME=/u01/app/19.25.0/grid
[grid@node1]$ cd $NEWGRIDHOME
[grid@node1]$ unzip -oq /u01/software/LINUX.X64_193000_grid_home.zip

Optionally, I can use a golden image.

I update OPatch to the latest version:

[grid@node1]$ cd $NEWGRIDHOME
[grid@node1]$ rm -rf OPatch
[grid@node1]$ unzip -oq /u01/software/p6880880_190000_Linux-x86-64.zip

Then, I check the Oracle Grid Infrastructure prerequisites. I am good to go if the check doesn’t write any error messages to the console:
```
[grid@node1]$ export ORACLE_HOME=$NEWGRIDHOME
[grid@node1]$ $ORACLE_HOME/gridSetup.sh -executePrereqs -silent
```
I want to apply the 19.25 Release Update while I install the GI home. To do that, I must extract the patch file:
```
 [grid@node1]$ cd /u01/software
 [grid@node1]$ unzip -oq p36916690_190000_Linux-x86-64.zip -d 36916690
```
- The GI Release Update is a bundle patch consisting of:
  - OCW Release Update (patch 36917416)
  - Database Release Update (36912597)
  - ACFS Release Update (36917397)
  - Tomcat Release Update (36940756)
  - DBWLM Release Update (36758186)
- I will apply all of them.

Finally, I can install the new GI home:

The parameter -applyRU is the path to the OCW Release Update.
The parameter -applyOneOffs is a comma-separated list of the paths to each of the other Release Updates in the GI bundle patch.
The environment variable CLUSTER_NAME is the name of my Oracle Restart stack.

[grid@node1]$ export ORACLE_BASE=/u01/app/grid
[grid@node1]$ export ORA_INVENTORY=/u01/app/oraInventory
[grid@node1]$ export ORACLE_HOME=$NEWGRIDHOME
[grid@node1]$ cd $ORACLE_HOME
[grid@node1]$ ./gridSetup.sh -ignorePrereq -waitforcompletion -silent \
   -applyRU /u01/software/36916690/36916690/36917416 \
   -applyOneOffs /u01/software/36916690/36916690/36912597,/u01/software/36916690/36916690/36917397,/u01/software/36916690/36916690/36940756,/u01/software/36916690/36916690/36758186 \ \   
   -responseFile $ORACLE_HOME/install/response/gridsetup.rsp \
   INVENTORY_LOCATION=$ORA_INVENTORY \
   ORACLE_BASE=$ORACLE_BASE \
   SELECTED_LANGUAGES=en \
   oracle.install.option=CRS_SWONLY \
   oracle.install.asm.OSDBA=asmdba \
   oracle.install.asm.OSOPER=asmoper \
   oracle.install.asm.OSASM=asmadmin \
   oracle.install.crs.config.ClusterConfiguration=STANDALONE \
   oracle.install.crs.config.configureAsExtendedCluster=false \oracle.install.crs.config.gpnp.configureGNS=false \
   oracle.install.crs.config.autoConfigureClusterNodeVIP=false

Although the script says so, I don’t run root.sh.
I install it in silent mode, but I could use the wizard instead.
You need to install the new GI home in a way that matches your environment.
For inspiration, you can check the response file used in the previous GI home on setting the various parameters.
If I have additional one-off patches to install, I add them to the comma-separated list.

2. Prepare a New Database Home

I can do this in advance. It doesn’t affect my current environment and doesn’t cause any downtime.

I need to create a folder for the new database home. I must do this as oracle:

[oracle@node1]$ export NEW_ORACLE_HOME=/u01/app/oracle/product/dbhome_1925
[oracle@node1]$ mkdir -p $NEW_ORACLE_HOME

I extract the base release of Oracle Database into the new database home:
```
[oracle@node1]$ cd $NEW_ORACLE_HOME
[oracle@node1]$ unzip -oq /u01/software/LINUX.X64_193000_db_home.zip
```
Optionally, I can use a golden image.

I update OPatch to the latest version:

[oracle@node1]$ rm -rf OPatch
[oracle@node1]$ unzip -oq /u01/software/p6880880_190000_Linux-x86-64.zip

I want to apply the 19.25 Database Release Update. In addition, I must also apply the OCW Release Update to the database home. I take those from the GI Release Update that I used earlier. In addition, I want to apply the OJVM and Data Pump bundle patches. Those I must extract.
```
[oracle@node1]$ cd /u01/software
[oracle@node1]$ unzip -oq p36878697_190000_Linux-x86-64.zip -d 36878697
[oracle@node1]$ unzip -oq p37056207_1925000DBRU_Generic_1925.zip -d 37056207   
```

Then, I can install the new database home and apply the patches at the same time:

The parameter -applyRU is the path to the Database Release Update.
The parameter -applyOneOffs is a comma-separated list of the paths to the OCW Release Update plus OJVM and Data Pump bundle patches.

[oracle@node1]$ export ORACLE_BASE=/u01/app/grid
[oracle@node1]$ export ORA_INVENTORY=/u01/app/oraInventory
[oracle@node1]$ export OLD_ORACLE_HOME=$ORACLE_HOME
[oracle@node1]$ export ORACLE_HOME=$NEW_ORACLE_HOME
[oracle@node1]$ cd $ORACLE_HOME
[oracle@node1]$ ./runInstaller -ignorePrereqFailure -waitforcompletion -silent \
     -responseFile $ORACLE_HOME/install/response/db_install.rsp \
     -applyRU /u01/software/36916690/36916690/36912597 \
     -applyOneOffs /u01/software/36916690/36916690/36917416,/u01/software/36878697/36878697,/u01/software/37056207/37056207 \
     oracle.install.option=INSTALL_DB_SWONLY \
     UNIX_GROUP_NAME=oinstall \
     INVENTORY_LOCATION=$ORA_INVENTORY \
     SELECTED_LANGUAGES=en \
     ORACLE_HOME=$ORACLE_HOME \
     ORACLE_BASE=$ORACLE_BASE \
     oracle.install.db.InstallEdition=EE \
     oracle.install.db.OSDBA_GROUP=dba \
     oracle.install.db.OSBACKUPDBA_GROUP=dba \
     oracle.install.db.OSDGDBA_GROUP=dba \
     oracle.install.db.OSKMDBA_GROUP=dba \
     oracle.install.db.OSRACDBA_GROUP=dba \
     oracle.install.db.isRACOneInstall=false \
     oracle.install.db.rac.serverpoolCardinality=0 \
     SECURITY_UPDATES_VIA_MYORACLESUPPORT=false \
     DECLINE_SECURITY_UPDATES=true

I install it in silent mode, but I could use the wizard instead.
You need to install the new database home in a way that matches your environment.
For inspiration, you can check the response file used in the previous database home on setting the various parameters.
If I have additional one-off patches to install, I add them to the comma-separated list.

I run the database root script:
```
[root@node1]$ $NEW_ORACLE_HOME/root.sh
```
- I run just the database root script. Not the GI root script.

3. Prepare Database

I can do this in advance. It doesn’t affect my current environment and doesn’t cause any downtime.

I will move the database into a new Oracle home, so I need to ensure the database configuration files are either outside the Oracle home or move them to the new Oracle home.

I verify that my SP file and password file are stored in ASM – or at least outside the Oracle home:
```
[oracle@node1]$ export ORACLE_HOME=$OLD_ORACLE_HOME
[oracle@node1]$ srvctl config database -db $ORACLE_UNQNAME | grep file  
```
- If the files are stored in the dbs folder, I copy them to new Oracle home.

I copy tnsnames.ora and sqlnet.ora to the new Oracle home:

[oracle@node1]$ cp $OLD_ORACLE_HOME/network/admin/sqlnet.ora $NEW_ORACLE_HOME/network/admin
[oracle@node1]$ cp $OLD_ORACLE_HOME/network/admin/tnsnames.ora $NEW_ORACLE_HOME/network/admin

I take care of any other configuration files in the Oracle home.

I modify the database so it starts in the new Oracle home on the next restart.

[oracle@node1]$ srvctl modify database -d $ORACLE_UNQNAME -o $NEW_ORACLE_HOME

4. Switch to the New GI and Database Homes

Now, I can complete the patching process by switching to the new Oracle homes.

I connect as root and start the switch:

[root@node1]$ export ORACLE_HOME=/u01/app/19.25.0/grid
[root@node1]$ $ORACLE_HOME/rdbms/install/rootadd_rdbms.sh
[root@node1]$ $ORACLE_HOME/crs/install/roothas.sh -prepatch -dstcrshome $ORACLE_HOME

Downtime starts now!
Then, I complete the switch.
- This step stops the entire GI stack, including resources it manages (databases, listener, etc.).
- Everything is restarted in the new Oracle homes.
```
[root@node1]$ $ORACLE_HOME/crs/install/roothas.sh -postpatch -dstcrshome $ORACLE_HOME
```
Downtime ends now. Users may connect to the database.

As grid, I update the inventory, so the new GI home is registered as the active one:

[grid@node1]$ export OLD_ORACLE_HOME=/u01/app/19.24.0/grid
[grid@node1]$ export NEW_ORACLE_HOME=/u01/app/19.25.0/grid
[grid@node1]$ $NEW_ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$NEW_ORACLE_HOME CRS=TRUE
[grid@node1]$ $OLD_ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$OLD_ORACLE_HOME CRS=FALSE

I update any profiles (e.g., .bash_profile) and other scripts referring to the GI home.
As oracle, I update any profiles (e.g., .bash_profile) and other scripts referring to the database home.

5. Complete Patching

I complete patching of the database by running Datapatch (ensure the environment is set correctly):
```
[oracle@node1]$ env | grep ORA
[oracle@node1]$ $ORACLE_HOME/OPatch/datapatch
```

Most likely, there are other changes that you need to make in your own environment:

Update Enterprise Manager registration
Upgrade RMAN catalog
Update other scripts
Update /etc/oratab

That’s it! I have now patched my Oracle Restart deployment.

Happy Patching!

Appendix

Deinstall

In the future, I should remove the old Oracle homes. I use the deinstall tool in the respective Oracle homes.

I would recommend waiting a week or two until I’m confident the new Release Updates are fine.

CRS-0245: User doesn’t have enough privilege to perform the operation

If you get the following error:

[oracle@node1]$ srvctl modify database -d $ORACLE_UNQNAME -o $NEW_ORACLE_HOME
PRCD-1163 : Failed to modify database DB19
PRCR-1071 : Failed to register or update resource ora.db19.db
CRS-0245:  User doesn't have enough privilege to perform the operation

Be sure to include patch 29326865 in GI and database home.
Run the srvctl modify database command as grid instead.
Be sure that the Oracle user is still set to oracle after running the command as grid:
```
[oracle@node1]$ srvctl config database -db $ORACLE_UNQNAME | grep user
```

Rollback

If you need to roll back, you more or less reverse the process. The switch home method works to a newer and lower patch level.

OCW Release Update

Thanks to Jan for commenting on the blog post. The initial version didn’t include the OCW Release Update into the database home, which is needed when the database is managed by Grid Infrastructure in any way.

Incorrect Information in `ocr.loc`

In the ocr.loc file for Oracle Restart, only the local_only property is used. All other properties can be ignored (like ocrconfig_loc).

In Oracle Database 23ai, the file will be cleaner in Oracle Restart. But for Oracle Database 19c there will be these superfluous properties.

How to Patch Oracle Database With One Command

With the latest release of Oracle AutoUpgrade, patching an Oracle Database is much easier. Once I’ve configured AutoUpgrade, it takes just one command. That includes finding and downloading the right patches and creating a new Oracle home.

Let’s see how I can do it.

Configuration

Get the latest version of AutoUpgrade:

wget https://download.oracle.com/otn-pub/otn_software/autoupgrade.jar

I need to create a config file:
- global.global_log_dir is the logging and working directory of AutoUpgrade.
- global.keystore is where AutoUpgrade can create a software keystore that stores my My Oracle Support credentials.
- sid and source_home describe the database that I want to patch.
- target_home is the location of the new Oracle home that AutoUpgrade creates for me.
- folder is the location where AutoUpgrade can find or download patches.
- patch tells AutoUpgrade to apply the latest Release Update including the OJVM and Data Pump bundle patches, plus update OPatch.

global.global_log_dir=/home/oracle/autoupgrade-patching/log
global.keystore=/home/oracle/autoupgrade-patching/keystore
patch1.source_home=/u01/app/oracle/product/19
patch1.target_home=/u01/app/oracle/product/19_25_0
patch1.sid=FTEX
patch1.folder=/home/oracle/autoupgrade-patching/patch
patch1.patch=RECOMMENDED

Currently, AutoUpgrade can’t download the Oracle Database 19c base release (LINUX.X64_193000_db_home.zip). I must do that manually and place it in the folder specified by the config file entry folder. I expect the next version to handle it automatically.

$ cd /home/oracle/autoupgrade-patching/patch
$ ls -l
total 2987996
-rw-r--r--. 1 oracle oinstall 3059705302 Mar 18  2020 LINUX.X64_193000_db_home.zip

I need to load my My Oracle Support credentials into the AutoUpgrade keystore so AutoUpgrade can use them to download patches. The first time AutoUpgrade creates the keystore, and I must provide a keystore password:

$ java -jar autoupgrade.jar -config FTEX.cfg -patch -load_password
Processing config file ...

Starting AutoUpgrade Patching Password Loader - Type help for available options
Creating new AutoUpgrade Patching keystore - Password required
Enter password:
Enter password again:
AutoUpgrade Patching keystore was successfully created

I add my credentials. This is the username and password that I use to connect to My Oracle Support:

MOS> add -user daniel.overby.hansen@oracle.com
Enter your secret/Password:
Re-enter your secret/Password:

I save the keystore and convert it to an auto-login keystore:

MOS> save
Convert the AutoUpgrade Patching keystore to auto-login [YES|NO] ? YES

I exit the keystore.

MOS> exit

AutoUpgrade Patching Password Loader finished - Exiting AutoUpgrade Patching

Patch

With one command, I will patch my Oracle Database:

$ java -jar autoupgrade.jar -config FTEX.cfg -patch -mode deploy
AutoUpgrade Patching 24.7.241021 launched with default internal options
Processing config file ...
Loading AutoUpgrade Patching keystore
AutoUpgrade Patching keystore was successfully loaded

Connected to MOS - Searching for specified patches

------------------------------------------------------------
Downloading files to /home/oracle/autoupgrade-patching/patch
------------------------------------------------------------
DATABASE RELEASE UPDATE 19.25.0.0.0
    File: p36912597_190000_Linux-x86-64.zip - VALIDATED

DATAPUMP BUNDLE PATCH 19.25.0.0.0
    File: p37056207_1925000DBRU_Generic.zip - VALIDATED

OJVM RELEASE UPDATE 19.25.0.0.0
    File: p36878697_190000_Linux-x86-64.zip - VALIDATED

OPatch 12.2.0.1.44 for DB 19.0.0.0.0 (Oct 2024)
    File: p6880880_190000_Linux-x86-64.zip - VALIDATED
------------------------------------------------------------

+-----------------------------------------+
| Starting AutoUpgrade Patching execution |
+-----------------------------------------+
1 Non-CDB(s) will be processed
Type 'help' to list console commands
patch> Job 100 completed
------------------- Final Summary --------------------
Number of databases            [ 1 ]

Jobs finished                  [1]
Jobs failed                    [0]
Jobs restored                  [0]
Jobs pending                   [0]

---- Drop GRP at your convenience once you consider it is no longer needed ----
Drop GRP from FTEX: drop restore point AU_PATCHING_9212_FTEX1921000


Please check the summary report at:
/home/oracle/autoupgrade-patching/log/cfgtoollogs/patch/auto/status/status.html
/home/oracle/autoupgrade-patching/log/cfgtoollogs/patch/auto/status/status.log

That’s it! My database is now running on Oracle Database 19.25.0.

Fine Print

AutoUpgrade does out-of-place patching according to our recommendations. It will use a brand-new Oracle home, and copies all configuration files from the old to the new Oracle home.

AutoUpgrade currently requires:

Oracle Database 19c
Single instance database (non-CDB or entire CDB)

If you need more details, please check out the webinar One-Button Patching with AutoUpgrade. The slide deck has additional details.

What About

This is the first release, and naturally, it comes with restrictions. We’re working hard behind the scenes to lift those requirements.

Will AutoUpgrade support other platforms? Yes. AutoUpgrade supports most of the platforms where you run Oracle Database, including Windows, AIX and SPARCH Solaris.

What about in-place patching, using cloned Oracle homes, and support for gold images? It’s in the plans.

Will you support other Oracle Database releases? Yes – anything newer than Oracle Database 19c

What about Data Guard and RAC? You can use AutoUpgrade for Data Guard. RAC is in the plans.

What about Grid Infrastructure? We’re focusing on Oracle Database. You should have a look at Oracle Fleet Patching and Provisioning.

What about … We’re looking for new, great ideas. Please create an enhancement request in My Oracle Support and send me the number. I’ll check it and add it to our backlog. Feel free to also send me an email with your suggestion.

Happy Patching!

It’s a Wrap – MakeIT 2024

In the last days of May, I had the absolute pleasure of attending the MakeIT 2024 conference. Once again, the conference was held in the beautiful seaside city of Portorož, Slovenia.

The conference is a joint venture with the JCON conference, so you can catch one or two sessions on Java as well. I really value learning about related technologies as well. This year I listened in on caching options in Java and the use of JDBC driver.

You should come next year. There is also a good selection of English speakers, so even for foreigners, there is a reason to travel.

The Slides

Workshop – Patching Oracle Database

This is a full workshop on patching Oracle Database. It includes a hands-on lab, so you can try it on your own.

You should flip through the slides if you want a deep dive on patching.

You can also try the hands-on lab for free.

Best Practices for Upgrade to Oracle Database 23ai

This session and the slides help you prepare for the next long-term support release of Oracle Database.

Patch Me If You Can

This session and the slides give a quick fly-over of the essentials of patching.

Oracle Data Pump – News, Internals, Tips and Tricks

I had the pleasure of talking about Oracle Data Pump and presenting some new features. If you’re curious about a faster way of creating indexes and adding constraints, you can flip through the slides.

Thanks

Thanks to the organizer of MakeIT 2024 for pulling off yet another successful conference, to the sponsors for making it all possible, and to everyone who attended my sessions or the conference in general.

Impressions

Quote of the conference

My Data Pump talk

Going to conference is hard

Low hanging clouds at the airport

How to Perform Standby-first Patch Apply When You Have Different Primary and Standby Databases in the Same Oracle Home

I am a big fan of Oracle Data Guard Standby-First Patch Apply. You can:

Reduce downtime to the time it takes to perform a switchover.
Test the patching procedure on the standby database.

I received an interesting question the other day:

I have the following two Data Guard configurations. I want to patch all the databases using standby-first patch apply. How do I do that when I have primary and standby databases running out of the same Oracle home on the same machine?

Requirements

In this case, the databases are on 19.17.0, and the customer wants to patch them to 19.23.0.

To use standby-first patch apply, you must meet a set of requirements, one being:

Data Guard Standby-First Patch Apply is supported between database patch releases that are a maximum of one year (1 year) apart based on the patch release date.

Here are the release dates of the following Release Updates:

19.17.0: October 2022
19.23.0: April 2024

So, in this case, the customer can’t use standby-first patch apply directly. There is a year and a half in between. They need to patch cycles in this case:

Patch to 19.21.0 (release October 2023)
Patch to 19.23.0 (release April 2024)

In the future, they should apply patches more often to avoid ending up in this situation again.

Patching Oracle Home

The customer has one Oracle home on each server from where both databases run. On any server, there is a primary and a standby database (from two different Data Guard configs).

The customer uses in-place patching. If they patch the entire Oracle home, it means one of the primary databases is now on a higher Oracle home than its standby database, which is not allowed. The standby database is the only one which may run on a higher patch level.

Using the above configuration with primary and standby databases running out of the same Oracle home, you can’t use in-place patching and standby-first patch apply.

The customer must switch to out-of-place patching to achieve this. Then you can patch standby databases first, then the primaries.

Plus, you get all the other benefits of out-of-place patching.

Datapatch

Once all the databases in a Data Guard configuration run in the new Oracle home, you still haven’t completed the patching process:

A patch or patch bundle is not considered fully installed until all of the following actions have occurred:

Patch binary installation has been performed to the database home on all standby systems.

Patch binary installation has been performed to the database home on the primary system.

Patch SQL installation, if required by the patch, has been performed on the primary database and the redo applied to the standby database(s).

You must do the above steps in the specified order, and the last step is to execute Datapatch:

$ $ORACLE_HOME/OPatch/datapatch

Step-by-step

You can use AutoUpgrade to patch Oracle Data Guard.

Happy Patching!

It’s a Wrap – UKOUG Conference ’23

I just finished my presentation at the UKOUG conference. This time, it was held at the Oracle office in Reading. Two intense days full of learning experiences.

It’s the 40th anniversary of UKOUG – that’s truly amazing. The community started when I was just a little child and still lives on today, what a change tech has undergone since then.

Congratulations to the board and the entire community on the 40th anniversary.

The Slides

Patch Me If You Can – Grid Infrastructure Edition

This is a modification of an existing talk about database patching, but mostly on Oracle Grid Infrastructure. But since Oracle Database and Grid Infrastructure go hand in hand, it also has some database stuff.

You should flip through the slides if you work with Oracle Grid Infrastructure. And remember – always patch out-of-place.

Help! My Database is still on 8i!

I also had the opportunity to close the conference with my 8i talk. I really like this talk because it is a walk down memory lane. Plus, it includes demos using Oracle 8i Database. It’s cool to be old school.

For a little laugh, you can find a comparison of Oracle Database releases and mobile phones of the same age.

Thanks

Thanks to the board of UKOUG and the organizers for pulling yet another successful conference. Thanks to the sponsors making it all possible and to everyone who attended my sessions or the conference in general.

It keeps impressing me how much you can learn in such a short time. My head is full. Luckily, the weekend is coming up.

P.S. The chocolate fountain was amazing (see below)!

Pictures

How to Clone Oracle Grid Infrastructure Home Using Golden Images

Cloning Oracle Grid Infrastructure (GI) homes is a convenient way of getting a new GI Home. It’s particularly helpful when you need to patch out-of-place using the SwitchGridHome method.

When you have created a new GI home and applied all the necessary patches, you can turn it into a golden image. Later on, you can deploy from that golden image and avoid updating OPatch and apply patches.

How to Create a Golden Image

First, only create a golden image from a freshly installed Oracle Home. Never use an Oracle Home that is already in use. As soon as you start to use an Oracle Home you taint it with various files and you don’t want to carry those files around in your golden image. The golden image must be completely clean.

Then, you create a directory where you can store the golden image:

export GOLDIMAGEDIR=/u01/app/grid/goldimages
mkdir -p $GOLDIMAGEDIR

Finally, you create the golden image. This command creates a golden image of the specified GI home:
```
export NEW_GRID_HOME=/u01/app/19.20.0/grid
$NEW_GRID_HOME/gridSetup.sh -createGoldImage \
   -destinationLocation $GOLDIMAGEDIR \
   -silent
```
Be sure to do this before you start to use the new GI home.
The installer creates the golden image as a zip file in the specified directory. The name of the zip file is unique and printed on the console. You can also use the secret parameter -name to specify a name for the zip file. To name the zip file gi_19_20_0.zip:
```
$NEW_GRID_HOME/gridSetup.sh -createGoldImage \
   ... \
   -name gi_19_20_0.zip
```

No software must run out of the Oracle Home, when you create the gold image. Don’t use a production Oracle Home. I recommend using a test or staging server instead.

Check the documentation for further details.

How to Deploy from a Golden Image

You must create a folder for the new GI home. You do it as root:

export NEW_GRID_BASE=/u01/app/19.20.0
export NEW_GRID_HOME=$NEW_GRID_BASE/grid
mkdir -p $NEW_GRID_HOME
chown -R grid:oinstall $NEW_GRID_BASE
chmod -R 775 $NEW_GRID_BASE

If you install the new GI home in a cluster, you must create the folder on all nodes.

Then, you extract the golden image as grid:

export NEW_GRID_HOME=/u01/app/19.20.0/grid
cd $NEW_GRID_HOME
unzip -q /u01/app/grid/goldimages/gi_19_20_0.zip

Finally, you use gridSetup.sh to perform the installation:
```
./gridSetup.sh 
```

That’s it!

I recommend using golden images when you patch out-of-place using the SwitchGridHome method.

Appendix

Oracle Restart vs. Oracle RAC

If you create a GI home for use with Oracle RAC, you can’t use that gold image for a new GI home for Oracle Restart.

Such two GI homes would be very different. You must have two gold images. One for RAC and one for Restart.

How to Roll Back Oracle Grid Infrastructure 19c Using SwitchGridHome

Let me show you how I roll back a patch from Oracle Grid Infrastructure 19c (GI) using the out-of-place method and the -switchGridHome parameter.

My demo system:

Is a 2-node RAC (nodes copenhagen1 and copenhagen2).
Runs Oracle Linux.
Was patched from 19.17.0 to 19.19.0. I patched both GI and database. Now I want GI back on 19.17.0.

I only roll back the GI home. See the appendix for a few thoughts on rolling back the database as well.

This method works if you applied the patch out-of-place – regardless of whether you used the OPatchAuto or SwitchGridHome method.

Preparation

I use the term old Oracle Home for the original, lower patch level Oracle Home.
- It is my 19.17.0 Oracle Home
- It is stored in /u01/app/19.0.0.0/grid
- I refer to this home using the environment variable OLD_ORACLE_HOME
- This is the Oracle Home that I want to roll back to
I use the term new Oracle Home for the higher patch level Oracle Home.
- It is my 19.19.0 Oracle Home
- It is stored in /u01/app/19.19.0/grid
- I refer to this home using the environment variable NEW_ORACLE_HOME
- This is the Oracle Home that I want to roll back from

Both GI homes are present in the system already.

How to Roll Back Oracle Grid Infrastructure 19c

1. Sanity Checks

I execute the following checks on both nodes, copenhagen1 and copenhagen2. I show the commands for one node only.

I verify that the active GI home is the new GI home:

[grid@copenhagen1]$ export ORACLE_HOME=$NEW_GRID_HOME
[grid@copenhagen1]$ $ORACLE_HOME/srvm/admin/getcrshome

I verify that the cluster upgrade state is NORMAL:

[grid@copenhagen1]$ $ORACLE_HOME/bin/crsctl query crs activeversion -f

I verify all CRS services are online:

[grid@copenhagen1]$ $ORACLE_HOME/bin/crsctl check cluster

I verify that the cluster patch level is 19.19.0 – the new patch level:
```
[grid@copenhagen1]$ $ORACLE_HOME/bin/crsctl query crs releasepatch
```

2. Cluster Verification Utility

I use Cluster Verification Utility (CVU) to verify that my cluster meets all prerequisites for a patch/rollback. I do this on one node only:
```
[grid@copenhagen1]$ $CVU_HOME/bin/cluvfy stage -pre patch
```
- You can find CVU in the GI home, but I recommend always getting the latest version from My Oracle Support.

3. Roll Back Node 1

The GI stack (including database, listener, etc.) needs to restart on each instance. But I do the rollback in a rolling manner, so the database stays up all the time.

I drain connections from the first node, copenhagen1.
I unlock the old GI home, root:
```
[root@copenhagen1]$ export OLD_GRID_HOME=/u01/app/19.0.0.0/grid
[root@copenhagen1]$ cd $OLD_GRID_HOME/crs/install
[root@copenhagen1]$ ./rootcrs.sh -unlock -crshome $OLD_GRID_HOME
```
- This is required because the next step (gridSetup.sh) runs as grid and must have access to the GI home.
- Later on, when I run root.sh, the script will lock the GI home.

I switch to old GI home as grid:

[grid@copenhagen1]$ export OLD_GRID_HOME=/u01/app/19.0.0.0/grid
[grid@copenhagen1]$ export ORACLE_HOME=$OLD_GRID_HOME
[grid@copenhagen1]$ export CURRENT_NODE=$(hostname)
[grid@copenhagen1]$ $ORACLE_HOME/gridSetup.sh \
   -silent -switchGridHome \
   oracle.install.option=CRS_SWONLY \
   ORACLE_HOME=$ORACLE_HOME \
   oracle.install.crs.config.clusterNodes=$CURRENT_NODE \
   oracle.install.crs.rootconfig.executeRootScript=false

I complete the switch by running root.sh as root:
```
[root@copenhagen1]$ export OLD_GRID_HOME=/u01/app/19.0.0.0/grid
[root@copenhagen1]$ $OLD_GRID_HOME/root.sh
```
- This step restarts the entire GI stack, including resources it manages (databases, listener, etc.). This means downtime on this node only. The remaining nodes stay up.
- In that period, GI marks the services as OFFLINE so users can connect to other nodes.
- If my database listener runs out of the Grid Home, GI will move it to the new Grid Home, including copying listener.ora.
- In the end, GI restarts the resources (databases and the like).
I update any profiles (e.g., .bashrc) and other scripts referring to the GI home.

I verify that the active GI home is the new GI home:

[grid@copenhagen1]$ $OLD_ORACLE_HOME/srvm/admin/getcrshome

I verify that the cluster upgrade state is ROLLING PATCH:

[grid@copenhagen1]$ $OLD_ORACLE_HOME/bin/crsctl query crs activeversion -f

I verify all CRS services are online:

[grid@copenhagen1]$ $OLD_ORACLE_HOME/bin/crsctl check cluster

I verify all resources are online:

[grid@copenhagen1]$ $OLD_ORACLE_HOME/bin/crsctl stat resource -t

I verify that the GI patch level is 19.17.0 – the old patch level:

[grid@copenhagen1]$ $OLD_ORACLE_HOME/bin/crsctl query crs releasepatch

4. Roll Back Node 2

I roll back the second node, copenhagen2, using the same process as the first node, copenhagen1.
- I double-check that the CURRENT_NODE environment variable gets updated to copenhagen2.
- When I use crsctl query crs activeversion -f to check the cluster upgrade state, it will now be back in NORMAL mode, because copenhagen2 is the last node in the cluster.

5. Cluster Verification Utility

I use Cluster Verification Utility (CVU) again. Now I perform a post-rollback check. I do this on one node only:
```
[grid@copenhagen1]$ $CVU_HOME/bin/cluvfy stage -post patch
```

That’s it!

My cluster is now operating at the previous patch level.

Appendix

SwitchGridHome Does Not Have Dedicated Rollback Functionality

OPatchAuto has dedicated rollback functionality that will revert the previous patch operation. Similar functionality does not exist when you use the SwitchGridHome method.

This is described in Steps for Minimal Downtime Grid Infrastructure Out of Place ( OOP ) Patching using gridSetup.sh (Doc ID 2662762.1). To rollback, simply switch back to the previous GI home using the same method as for the patch.

There is no real rollback option as this is a switch from OLD_HOME to NEW_HOME To return to the old version you need to recreate another new home and switch to that.

Should I Roll Back the Database as Well?

This post describes rolling back the GI home only. Usually, I recommend keeping the database and GI patch level in sync. If I roll back GI, should I also roll back the database?

The short answer is no!

Keeping the GI and database patch in sync is a good idea. But when you need to roll back, you are in a contingency. Only roll back the component that gives you problems. Then, you will be out of sync for a period of time until you can get a one-off patch or move to the next Release Update. Being in this state for a shorter period is perfectly fine – and supported.

Files to Move During Oracle Grid Infrastructure Out-Of-Place Patching

Like with Oracle Database, I strongly recommend patching Oracle Grid Infrastructure using the out-of-place method. It has many advantages over in-place patching

A while ago, I wrote about the files you must move during Oracle Database out-of-place patching. There are quite many files to consider. A blog reader left a comment asking about a similar blog post for Oracle Grid Infrastructure, so here it is.

Listener

The listener runs out of the GI home. When you patch out-of-place, the listener must also start in the new GI home.

If you patch using any of the below methods, it will move the listener for you:

OPatchAuto – when you execute opatchauto apply ... -switch-clone.
SwitchGridHome – when you run root.sh.

The two methods also move listener.ora as part of the process.

If you use opatchauto you should note that the tool moves listener.ora when preparing the new GI using opatchauto apply ... -prepare-clone. You can run that command hours or days before you move the listener. If you add things to listener.ora in between, you must also add it to listener.ora in the new GI home.

Conclusion

There is really nothing to worry about when you patch Oracle Grid Infrastructure out-of-place. The above-mentioned two tools will take care of it for you.

Happy Patching

A Rare Insight Into the Oracle Database Development Process

Have you ever wondered why Oracle didn’t include your bug fix in the next Release Update? Or what good is it that your bug is fixed in 23.1 when Oracle Database isn’t released yet?

We’ll explain this – and much more in our next webinar.

Episode 17: From SR to Patch June 22, 2023, 16:00 CEST

Mike and I will show you what happens behind the scenes, from opening a service request to the final delivery of a fix. This is your rare chance to get insights into the Oracle Database development process from insiders. And even if you are a long-time Oracle expert, you will still learn something new.

You can sign up here.

But I Can’t Make It

Don’t worry. As usual, we will publish the recording on our YouTube channel and share the slides with you. Keep an eye out on my Webinars page. On the same page, you can also watch all previous webinars and get the slides.

But it’s better to watch it live. You can ask questions to us live. I promise you; we won’t leave until we have answered all your questions.

All Tech, No Marketing

Remember, our mantra is: All tech, no marketing.

I hope to see you there!

How to Apply Patches Out-of-place to Oracle Grid Infrastructure and Oracle Data Guard Using Standby-First

I strongly recommend that you always patch out-of-place. Here’s an example of how to do it on Oracle Grid Infrastructure (GI) and Oracle Data Guard using Standby-First Patch Apply.

Standby-First Patch Apply allows you to minimize downtime to the time it takes to perform a Data Guard switchover. Further, it allows you to test the apply mechanism on the standby database by temporarily converting it into a snapshot standby database.

The scenario:

Oracle Grid Infrastructure 19c and Oracle Database 19c
Patching from Release Update 19.17.0 to 19.19.0
Vertical patching – GI and database at the same time
Data Guard setup with two RAC databases
- Cluster 1: copenhagen1 and copenhagen2
- Cluster 2: aarhus1 and aarhus2
- DB_NAME: CDB1
- DB_UNIQUE_NAME: CDB1_COPENHAGEN and CDB1_AARHUS
Using Data Guard broker
Patching GI using SwitchGridHome method

Let’s get started!

Step 1: Prepare

I can make the preparations without interrupting the database.

I ensure my environment meets the requirements for Standby-First Patch Apply.
I deploy new GI homes to all four hosts.
- I use the SwitchGridHome method.
- Very important: I only perform step 1 (Prepare a New Grid Home).
- I apply the Release Update 19.19.0 as part of the deployment using gridSetup.sh ... -applyRU ... -applyOneOffs as described in the blog post.
I deploy new database homes to all four hosts.
- I prefer to use brand-new database homes. If you clone an existing database home, be sure to clean it up before you proceed.
- When I attach the new database home to the inventory, I also apply the Release Update 19.19.0 using runInstaller ... -applyRU.
I also recompile invalid objects. This can make it easier for Datapatch later in the process:
```
PRIMARY SQL> @?/rdbms/admin/utlrp
```

Step 2: Restart Standby in New Oracle Homes

Now, I can move the standby database to the new GI and database homes.

On the standby hosts, aarhus1 and aarhus2, I first move the database configuration files from the old database home to the new one.

I change the database configuration in GI. Next time the database restarts, it will be in the new Oracle Home:

[oracle@aarhus1]$ $OLD_ORACLE_HOME/bin/srvctl modify database \
   -db $ORACLE_UNQNAME \
   -oraclehome $NEW_ORACLE_HOME

I switch to the new GI on all standby hosts, aarhus1 and aarhus2, by executing step 2 (Switch to the new Grid Home) of the SwitchGridHome method.
- It involves running gridSetup.sh ... -switchGridHome and root.sh.
- You can perform the switch in a rolling manner or all at once.
- The switch restarts the standby database instance. The standby database instance restarts in the new Oracle Home.
- If the profile of grid (like .bashrc) sets the ORACLE_HOME environment variable, I ensure to update it.
If I had multiple standby databases, I would process all standby databases in this step.

Step 3: Test Standby Database

This is an optional step, but I recommend that you do it.

I convert the standby database (CDB1_AARHUS) to a snapshot standby database:
```
DGMGRL> convert database CDB1_AARHUS to snapshot standby;
```
I test Datapatch on the standby database. It is important that I run the command on the standby database:
```
[oracle@aarhus1]$ $ORACLE_HOME/OPatch/datapatch -verbose
```
I can also test my application on the standby database.
At the end of my testing, I revert the standby database to a physical standby database. The database automatically reverts all the changes made during testing:
```
DGMGRL> convert database CDB1_AARHUS to physical standby;
```

Step 4: Switchover

I can perform the previous steps without interrupting my users. This step requires a maintenance window because I am doing a Data Guard switchover.

I check that my standby database is ready to become primary. Then, I start a Data Guard switchover:

DGMGRL> connect sys/<password> as sysdba
DGMGRL> validate database CDB1_AARHUS;
DGMGRL> switchover to CDB1_AARHUS;

A switchover does not have to mean downtime.

If my application is configured properly, the users will experience a brownout; a short hang, while the connections switch to the new primary database.

Step 5: Restart New Standby in New Oracle Homes

Now, the primary database runs on aarhus1 and aarhus2. Next, I can move the new standby hosts, copenhagen1 and copenhagen2, to the new GI and database homes.

I repeat step 2 (Restart Standby In New Oracle Homes) but this time for the new standby hosts, copenhagen1 and copenhagen2.

Step 6: Complete Patching

Now, both databases in my Data Guard configuration run out of the new Oracle Homes.

Only proceed with this step once all databases run out of the new Oracle Home.

I need to run this step as fast as possible after I have completed the previous step.

I complete the patching by running Datapatch on the primary database (CDB1_AARHUS). I add the recomp_threshold parameter to ensure Datapatch recompiles all objects that the patching invalidated:
```
[orale@aarhus1]$ $ORACLE_HOME/OPatch/datapatch \
   -verbose \
   -recomp_threshold 10000
```
- I only need to run Datapatch one time. On the primary database and only on one of the instances.
I can run Datapatch while users are connected to my database.
Optionally, I can switch back to the original primary database on copenhagen1 and copenhagen2, if I prefer to run it there.

That’s it. Happy patching!

My demo system

Preparation

How to Patch Oracle Restart 19c and Oracle Database

1. Prepare a New GI Home

2. Prepare a New Database Home

3. Prepare Database

4. Switch to the New GI and Database Homes

5. Complete Patching

Happy Patching!

Appendix

Deinstall

CRS-0245: User doesn’t have enough privilege to perform the operation

Rollback

OCW Release Update

Incorrect Information in ocr.loc

Further Reading

Other Blog Posts in This Series

Configuration

Patch

Fine Print

What About

Happy Patching!

The Slides

Workshop – Patching Oracle Database

Best Practices for Upgrade to Oracle Database 23ai

Patch Me If You Can

Oracle Data Pump – News, Internals, Tips and Tricks

Thanks

Impressions

Requirements

Patching Oracle Home

Datapatch

Step-by-step

Happy Patching!

The Slides

Patch Me If You Can – Grid Infrastructure Edition

Help! My Database is still on 8i!

Thanks

Pictures

How to Create a Golden Image

How to Deploy from a Golden Image

That’s it!

Appendix

Oracle Restart vs. Oracle RAC

Further Reading

Other Blog Posts in This Series

Preparation

How to Roll Back Oracle Grid Infrastructure 19c

1. Sanity Checks

2. Cluster Verification Utility

3. Roll Back Node 1

4. Roll Back Node 2

5. Cluster Verification Utility

That’s it!

Appendix

SwitchGridHome Does Not Have Dedicated Rollback Functionality

Should I Roll Back the Database as Well?

Other Blog Posts in This Series

Listener

Conclusion

Happy Patching

But I Can’t Make It

All Tech, No Marketing

Step 1: Prepare

Step 2: Restart Standby in New Oracle Homes

Step 3: Test Standby Database

Step 4: Switchover

Step 5: Restart New Standby in New Oracle Homes

Step 6: Complete Patching

Appendix

Further Reading

Other Blog Posts in This Series

Incorrect Information in `ocr.loc`