How to Patch Oracle Grid Infrastructure 19c Using Out-Of-Place SwitchGridHome

Let me show you how I patch Oracle Grid Infrastructure 19c (GI) using the out-of-place method and the -switchGridHome parameter.

My demo system:

Is a 2-node RAC
Runs Oracle Linux
Is currently on 19.16.0, and I want to patch to 19.17.0
Uses grid as the owner of the software

I patch only the GI home. If I want to patch the database as well, I must do it separately.

Preparation

I need to download:

Download the base release of Oracle Grid Infrastructure (LINUX.X64_193000_grid_home.zip) from oracle.com or Oracle Software Delivery Cloud.
Latest OPatch from My Oracle Support (6880880).
The 19.17.0 Release Update for Oracle Grid Infrastructure from My Oracle Support. I will use the combo patch (34449117).

I place the software in /u01/software.

How to Patch Oracle Grid Infrastructure 19c

1. Prepare a New Grid Home

I can do this in advance. It doesn’t affect my current environment and doesn’t cause any downtime.

I need to create a folder for the new Grid Home. I must do this as root on all nodes in my cluster (copenhagen1 and copenhagen 2):

[root@copenhagen1]$ mkdir -p /u01/app/19.17.0/grid
[root@copenhagen1]$ chown -R grid:oinstall /u01/app/19.17.0
[root@copenhagen1]$ chmod -R 775 /u01/app/19.17.0

[root@copenhagen2]$ mkdir -p /u01/app/19.17.0/grid
[root@copenhagen2]$ chown -R grid:oinstall /u01/app/19.17.0
[root@copenhagen2]$ chmod -R 775 /u01/app/19.17.0

I switch to the Grid Home owner, grid.
I ensure that there is passwordless SSH access between all the cluster nodes. It is a requirement for the installation, but sometimes it is disabled to strengthen security:
```
[grid@copenhagen1]$ ssh copenhagen2 date

[grid@copenhagen2]$ ssh copenhagen1 date
```

I extract the base release of Oracle Grid Infrastructure into the new Grid Home. I work on one node only:

[grid@copenhagen1]$ export NEWGRIDHOME=/u01/app/19.17.0/grid
[grid@copenhagen1]$ cd $NEWGRIDHOME
[grid@copenhagen1]$ unzip -oq /u01/software/LINUX.X64_193000_grid_home.zip

I update OPatch to the latest version:

[grid@copenhagen1]$ cd $NEWGRIDHOME
[grid@copenhagen1]$ rm -rf OPatch
[grid@copenhagen1]$ unzip -oq /u01/software/p6880880_190000_Linux-x86-64.zip

Then, I check the Oracle Grid Infrastructure prerequisites. I am good to go, if the check doesn’t write any error messages to the console:
```
[grid@copenhagen1]$ export ORACLE_HOME=$NEWGRIDHOME
[grid@copenhagen1]$ $ORACLE_HOME/gridSetup.sh -executePrereqs -silent
```

I want to apply the 19.17.0 Release Update while I install the Grid Home. To do that, I must extract the patch file:

 [grid@copenhagen1]$ cd /u01/software
 [grid@copenhagen1]$ mkdir 34449117
 [grid@copenhagen1]$ mv p34449117_190000_Linux-x86-64.zip 34449117
 [grid@copenhagen1]$ cd 34449117
 [grid@copenhagen1]$ unzip p34449117_190000_Linux-x86-64.zip

Finally, I can install the new Grid Home:

I need to update the environment variables.
CLUSTER_NODES is a comma-separated list of all the nodes in my cluster.
The parameter -applyRU must point to the directory holding the GI Release Update. Since I am using the combo patch, I need to specify the subdirectory containing the GI Release Update

[grid@copenhagen1]$ export ORACLE_BASE=/u01/app/grid
[grid@copenhagen1]$ export ORA_INVENTORY=/u01/app/oraInventory
[grid@copenhagen1]$ export CLUSTER_NAME=$(olsnodes -c)
[grid@copenhagen1]$ export CLUSTER_NODES=$(olsnodes | tr '\n' ','| sed 's/,\s*$//')
[grid@copenhagen1]$ cd $ORACLE_HOME
[grid@copenhagen1]$ ./gridSetup.sh -ignorePrereq -waitforcompletion -silent \
   -applyRU /u01/software/34449117/34449117/34416665 \
   -responseFile $ORACLE_HOME/install/response/gridsetup.rsp \
   INVENTORY_LOCATION=$ORA_INVENTORY \
   ORACLE_BASE=$ORACLE_BASE \
   SELECTED_LANGUAGES=en,en_GB \
   oracle.install.option=CRS_SWONLY \
   oracle.install.asm.OSDBA=asmdba \
   oracle.install.asm.OSOPER=asmoper \
   oracle.install.asm.OSASM=asmadmin \
   oracle.install.crs.config.ClusterConfiguration=STANDALONE \
   oracle.install.crs.config.configureAsExtendedCluster=false \
   oracle.install.crs.config.clusterName=$CLUSTER_NAME \
   oracle.install.crs.config.gpnp.configureGNS=false \
   oracle.install.crs.config.autoConfigureClusterNodeVIP=false \
   oracle.install.crs.config.clusterNodes=$CLUSTER_NODES

Although the script says so, I don’t run root.sh yet.
I install it in silent mode, but I could use the wizard instead.
For inspiration, you can check the response file used in the previous Grid Home on setting the various parameters.
If I have one-off patches to install, I can use the -applyOneOffs parameter.

2. Switch to the new Grid Home

Now, I can complete the patching process by switching to the new Grid Home. I do this one node at a time. Step 2 involves downtime.

First, on copenhagen1, I switch to the new Grid Home:

[grid@copenhagen1]$ export ORACLE_HOME=/u01/app/19.17.0/grid
[grid@copenhagen1]$ export CURRENT_NODE=$(hostname)
[grid@copenhagen1]$ $ORACLE_HOME/gridSetup.sh \
   -silent -switchGridHome \
   oracle.install.option=CRS_SWONLY \
   ORACLE_HOME=$ORACLE_HOME \
   oracle.install.crs.config.clusterNodes=$CURRENT_NODE \
   oracle.install.crs.rootconfig.executeRootScript=false

Then, I run the root.sh script as root:
- This step restarts the entire GI stack, including resources it manages (databases, listener, etc.). This means downtime on this node only. The remaining nodes stay up.
- In that period, GI marks the services as OFFLINE so users can connect to other nodes.
- If my database listener runs out of the Grid Home, GI will move it to the new Grid Home, including copying listener.ora.
- Optionally, if I want a more graceful approach, I can manually stop the services, and perform draining.
- In the end, GI restarts the resources (databases and the like).
```
[root@copenhagen1]$ /u01/app/19.17.0/grid/root.sh
```
I update any profiles (e.g., .bashrc) and other scripts referring to the Grid Home.
I connect to the other node, copenhagen2, and repeat steps 1-3. I double-check that the CURRENT_NODE environment variable gets updated to copenhagen2.

That’s it! I have now patched my Grid Infrastructure deployment.

Later on, I can patch my databases as well.

A Word about the Directory for the New Grid Home

Be careful when choosing a location for the new Grid Home. The documentation lists some requirements you should be aware of.

In my demo environment, the existing Grid Home is:

/u01/app/19.0.0.0/grid

Since I am patching to 19.17.0, I think it makes sense to use:

/u01/app/19.17.0/grid

If your organization has a different naming standard, that’s fine. Just ensure you comply with the requirements specified in the documentation.

Don’t Forget to Clean Your Room

At a future point, I need to remove the old Grid Home. I use the deinstall tool in the Grid Home. I execute the command on all nodes in my cluster:

$ export OLD_GRID_HOME=/u01/app/19.0.0.0/grid
$ export ORACLE_HOME=OLD_GRID_HOME
$ $ORACLE_HOME/deinstall/deinstall -local

I will wait until:

I have seen the new Grid Home run without problems for a week or two.
I have patched my Oracle Databases managed by GI.
I have seen my Oracle Databases run without GI-related problems for a week or two.

Happy Patching!

Appendix

Can I Run Datapatch When Users Are Connected

The short answer is: Yes! The longer answer is: Yes, but very busy systems or in certain situations, you might experience a few hiccups.

The obvious place to look for the answer would be in the documentation. Unfortunately, there is no Patching Guide similar to the Upgrade Guide. The information in this blog post is pieced together from many different sources.

A few facts about patching with Datapatch:

The database must be open in read write mode.
You can’t run Datapatch on a physical standby database – even if it’s open (Active Data Guard).
A patch is not fully installed until you have executed Datapatch successfully.

How To

First, let me state that it is fully supported to run Datapatch on a running database with users connected.

The procedure:

Install a new Oracle Home and use OPatch to apply the desired patches.
Shut down the database.
Restart the database in the new, patched Oracle Home.
Downtime is over! Users are allowed to connect to the database
Execute ./datapatch -verbose.
End of procedure. The patch is now fully applied.

Often users move step 3 to the end of the procedure. That’s of course also perfectly fine, but it does extend the downtime needed and often is not needed.

What About RAC and Data Guard

The above procedure is exactly what happens in a rolling patch apply on a RAC database. When you perform a rolling patch apply on a RAC database, there is no downtime at all. You use opatchauto to patch a RAC database. opatchauto restarts all instances of the database in the patched Oracle Home in a rolling manner. Finally, it executes datapatch on the last node. Individual instances are down temporarily, but the database is always up.

It is a similar situation when you use the Standby First Patch Apply. First, you restart all standby databases in the patched Oracle Home. Then, you perform a switchover and restart the former primary database in the patched Oracle Home. Finally, you execute datapatch to complete the patch installation. You must execute datapatch on the primary database.

Either way, don’t use Datapatch until all databases or instances run on the new, patched Oracle Home.

That’s It?

Yes, but I did write initally that there might be hiccups.

Waits

Datapatch connects to the database like any other session to make changes inside the database. These changes could be:

Creating new tables
Altering existing tables
Creating or altering views
Recreating PL/SQL packages like DBMS_STATS

Imagine this scenario:

Database is restarted in patched Oracle Home.
A user connects and starts to use DBMS_STATS.
You execute datapatch.
1. DBMS_STATS must be recreated to fix a bug.
2. Datapatch executes CREATE OR REPLACE PACKAGE SYS.DBMS_STATS .....
3. The Datapatch session will go into a wait.
User is done with DBMS_STATS.
The Datapatch session will come out of wait and replace the package.

In this scenario, the patching procedure was prolonged due to the wait. But it was completed eventually.

Hangs

From time to time, we are told that Datapatch hangs. Most likely, it is not a real hang, but just a wait on a lock. You can connect to the database and identify the blocker. You might even want to kill the blocking session to allow Datapatch to do its work.

Timeouts

What will happen in the above scenario if the user never releases the lock on DBMS_STATS? After a while, the DDL statement executed by Datapatch will error out:

ORA-04021: timeout occurred while waiting to lock object

To resolve this problem, restart Datapatch and ensure that there are no blocking sessions.

Really Busy Databases

I recommend patching at off-peak hours to reduce the likelihood of hitting the above problems.

If possible, you can also limit the activity in the database while you perform the patching. If your application is using e.g. DBMS_STATS and locking on that object is often a problem, you can hold off these sessions for a little while.

Similarly, if Advanced Queeing is causing problems, perhaps it helps temporarily set aq_tm_processes to 0. Or, in the case of the scheduler, job_queue_processes.

If nothing helps your situation, you can patch in restricted mode. But that means downtime:

SQL> startup restrict
./datapatch -verbose
SQL> alter system disable restricted session;

I don’t recommend starting in upgrade mode. To get out of upgrade mode a database restart is needed extending the downtime window.

Datapatch And Resources

How much resources does Datapatch need? Should I be worried about Datapatch depleting the system?

No, you should not. The changes that Datapatch needs to make are not resource-intensive. However, a consequence of the DDL statements might be object invalidation. But even here, you should not worry. Datapatch will automatically recompile any ORACLE_MAINTAINED object that was invalidated by the patch apply. But the recompilation happens serially, i.e., less resources needed.

Of course, if you system is running at 99% capacity, it might be a problem. On the other hand, if your system is at 99%, patching problems are probably the least of your worries.

What About OJVM

If you are using OJVM and you apply the OJVM bundle patch, things are a little different.

Release	RAC Rolling	Standby-First	Datapatch
Oracle Database 21c	Fully	No	No Datapatch downtime.
Oracle Database 19c + 18c	Partial	No	No Datapatch downtime, but java system is patched which requires ~10 second outage. Connected clients using java will receive ORA-29548.
Oracle Database 12.2 + 12.1	No	No	Datapatch must execute in upgrade mode.
Oracle Database 11.2.0.4	No	No	Similar to 12.2 and 12.1 except you don’t use Datapatch.

Mike Dietrich also has a good blog that you might want to read: Do you need STARTUP UPGRADE for OJVM?

What About Oracle GoldenGate

You should stop Oracle GoldenGate when you execute datapatch. When datapatch is done, you can restart Oracle GoldenGate.

If you are manually recompiling objects after datapatch, I recommend that you restart Oracle GoldenGate after the recompilation.

The above applies even if the patches being applied does not contain any Oracle GoldenGate specific patches.

Oracle GoldenGate uses several objects owned by SYS. When datapatch is running it might change some of those objects. In that case, unexpected errors might occur.

Recommendations

Before starting the patching procedure and downtime, I recommend you recompile invalid objects.
```
SQL> @?/rdbms/admin/utlrp
```
Always execute Datapatch with the -verbose flag. This will give you much better information about is going on.
```
$ $ORACLE_HOME/OPatch/datapatch -verbose
```
Always use the latest OPatch.
Always use out-of-place patching, even for RAC databases.

Conclusion

Go ahead and patch your database with Datapatch while users are connected.

Tag: opatch