How to Patch Oracle Restart 19c and Oracle Database Using Out-Of-Place Switch Home

Grid Infrastructure, Patching 8 Minutes

Let me show you how I patch Oracle Restart and Oracle Database 19c using the out-of-place method by switching to the new Oracle homes.

The advantages of this solution:

  • I get more control over the process
  • I can perform the entire operation with just one database restart
  • I can create my Oracle homes using gold images
  • I can prepare the new Oracle homes in advance
  • Overall, I find this method less riskier

My demo system

  • Single instance database in Oracle Restart configuration
  • Runs Oracle Linux
  • GI and database home are currently on 19.24

I want to:

  • patch to 19.25
  • patch both the GI and database home in one operation

Preparation

I need to download:

  1. The base releases of:
    • Oracle Grid Infrastructure (LINUX.X64_193000_grid_home.zip)
    • Oracle Database (LINUX.X64_193000_db_home.zip)
  2. Latest OPatch from My Oracle Support (6880880).
  3. Patches from My Oracle Support:
    • 19.25 Release Update for Grid Infrastructure (36916690)
    • Matching OJVM bundle patch (36878697)
    • Matching Data Pump bundle patch (36682332)

You can use AutoUpgrade to easily download GI patches.

I place the software in /u01/software.

How to Patch Oracle Restart 19c and Oracle Database

1. Prepare a New GI Home

I can do this in advance. It doesn’t affect my current environment and doesn’t cause any downtime.

  1. I need to create a folder for the new GI home. I must do this as root:

    [root@node1]$ mkdir -p /u01/app/19.25.0/grid
[root@node1]$ chown -R grid:oinstall /u01/app/19.25.0
[root@node1]$ chmod -R 775 /u01/app/19.25.0

  2. I switch to the GI home owner, grid.

  3. I extract the base release of Oracle Grid Infrastructure into the new GI home:

    [grid@node1]$ export OLDGRIDHOME=$ORACLE_HOME
[grid@node1]$ export NEWGRIDHOME=/u01/app/19.25.0/grid
[grid@node1]$ cd $NEWGRIDHOME
[grid@node1]$ unzip -oq /u01/software/LINUX.X64_193000_grid_home.zip

    Optionally, I can use a golden image.

  4. I update OPatch to the latest version:

    [grid@node1]$ cd $NEWGRIDHOME
[grid@node1]$ rm -rf OPatch
[grid@node1]$ unzip -oq /u01/software/p6880880_190000_Linux-x86-64.zip

  5. Then, I check the Oracle Grid Infrastructure prerequisites. I am good to go if the check doesn’t write any error messages to the console:

    [grid@node1]$ export ORACLE_HOME=$NEWGRIDHOME
[grid@node1]$ $ORACLE_HOME/gridSetup.sh -executePrereqs -silent

  6. I want to apply the 19.25 Release Update while I install the GI home. To do that, I must extract the patch file:

     [grid@node1]$ cd /u01/software
 [grid@node1]$ unzip -oq p36916690_190000_Linux-x86-64.zip -d 36916690
    • The GI Release Update is a bundle patch consisting of:
      • OCW Release Update (patch 36917416)
      • Database Release Update (36912597)
      • ACFS Release Update (36917397)
      • Tomcat Release Update (36940756)
      • DBWLM Release Update (36758186)
    • I will apply all of them.

  7. Finally, I can install the new GI home:

    • The parameter -applyRU is the path to the OCW Release Update.
    • The parameter -applyOneOffs is a comma-separated list of the paths to each of the other Release Updates in the GI bundle patch.
    • The environment variable CLUSTER_NAME is the name of my Oracle Restart stack.
    [grid@node1]$ export ORACLE_BASE=/u01/app/grid
[grid@node1]$ export ORA_INVENTORY=/u01/app/oraInventory
[grid@node1]$ export ORACLE_HOME=$NEWGRIDHOME
[grid@node1]$ cd $ORACLE_HOME
[grid@node1]$ ./gridSetup.sh -ignorePrereq -waitforcompletion -silent \
   -applyRU /u01/software/36916690/36916690/36917416 \
   -applyOneOffs /u01/software/36916690/36916690/36912597,/u01/software/36916690/36916690/36917397,/u01/software/36916690/36916690/36940756,/u01/software/36916690/36916690/36758186 \ \   
   -responseFile $ORACLE_HOME/install/response/gridsetup.rsp \
   INVENTORY_LOCATION=$ORA_INVENTORY \
   ORACLE_BASE=$ORACLE_BASE \
   SELECTED_LANGUAGES=en \
   oracle.install.option=CRS_SWONLY \
   oracle.install.asm.OSDBA=asmdba \
   oracle.install.asm.OSOPER=asmoper \
   oracle.install.asm.OSASM=asmadmin \
   oracle.install.crs.config.ClusterConfiguration=STANDALONE \
   oracle.install.crs.config.configureAsExtendedCluster=false \oracle.install.crs.config.gpnp.configureGNS=false \
   oracle.install.crs.config.autoConfigureClusterNodeVIP=false
    • Although the script says so, I don’t run root.sh.
    • I install it in silent mode, but I could use the wizard instead.
    • You need to install the new GI home in a way that matches your environment.
    • For inspiration, you can check the response file used in the previous GI home on setting the various parameters.
    • If I have additional one-off patches to install, I add them to the comma-separated list.

2. Prepare a New Database Home

I can do this in advance. It doesn’t affect my current environment and doesn’t cause any downtime.

  1. I need to create a folder for the new database home. I must do this as oracle:

    [oracle@node1]$ export NEW_ORACLE_HOME=/u01/app/oracle/product/dbhome_1925
[oracle@node1]$ mkdir -p $NEW_ORACLE_HOME

  2. I extract the base release of Oracle Database into the new database home:

    [oracle@node1]$ cd $NEW_ORACLE_HOME
[oracle@node1]$ unzip -oq /u01/software/LINUX.X64_193000_db_home.zip

    Optionally, I can use a golden image.

  3. I update OPatch to the latest version:

    [oracle@node1]$ rm -rf OPatch
[oracle@node1]$ unzip -oq /u01/software/p6880880_190000_Linux-x86-64.zip

  4. I want to apply the 19.25 Database Release Update. In addition, I must also apply the OCW Release Update to the database home. I take those from the GI Release Update that I used earlier. In addition, I want to apply the OJVM and Data Pump bundle patches. Those I must extract.

    [oracle@node1]$ cd /u01/software
[oracle@node1]$ unzip -oq p36878697_190000_Linux-x86-64.zip -d 36878697
[oracle@node1]$ unzip -oq p37056207_1925000DBRU_Generic_1925.zip -d 37056207

  5. Then, I can install the new database home and apply the patches at the same time:

    • The parameter -applyRU is the path to the Database Release Update.
    • The parameter -applyOneOffs is a comma-separated list of the paths to the OCW Release Update plus OJVM and Data Pump bundle patches.
    [oracle@node1]$ export ORACLE_BASE=/u01/app/grid
[oracle@node1]$ export ORA_INVENTORY=/u01/app/oraInventory
[oracle@node1]$ export OLD_ORACLE_HOME=$ORACLE_HOME
[oracle@node1]$ export ORACLE_HOME=$NEW_ORACLE_HOME
[oracle@node1]$ cd $ORACLE_HOME
[oracle@node1]$ ./runInstaller -ignorePrereqFailure -waitforcompletion -silent \
     -responseFile $ORACLE_HOME/install/response/db_install.rsp \
     -applyRU /u01/software/36916690/36916690/36912597 \
     -applyOneOffs /u01/software/36916690/36916690/36917416,/u01/software/36878697/36878697,/u01/software/37056207/37056207 \
     oracle.install.option=INSTALL_DB_SWONLY \
     UNIX_GROUP_NAME=oinstall \
     INVENTORY_LOCATION=$ORA_INVENTORY \
     SELECTED_LANGUAGES=en \
     ORACLE_HOME=$ORACLE_HOME \
     ORACLE_BASE=$ORACLE_BASE \
     oracle.install.db.InstallEdition=EE \
     oracle.install.db.OSDBA_GROUP=dba \
     oracle.install.db.OSBACKUPDBA_GROUP=dba \
     oracle.install.db.OSDGDBA_GROUP=dba \
     oracle.install.db.OSKMDBA_GROUP=dba \
     oracle.install.db.OSRACDBA_GROUP=dba \
     oracle.install.db.isRACOneInstall=false \
     oracle.install.db.rac.serverpoolCardinality=0 \
     SECURITY_UPDATES_VIA_MYORACLESUPPORT=false \
     DECLINE_SECURITY_UPDATES=true
    • I install it in silent mode, but I could use the wizard instead.
    • You need to install the new database home in a way that matches your environment.
    • For inspiration, you can check the response file used in the previous database home on setting the various parameters.
    • If I have additional one-off patches to install, I add them to the comma-separated list.

  6. I run the database root script:

    [root@node1]$ $NEW_ORACLE_HOME/root.sh
    • I run just the database root script. Not the GI root script.

3. Prepare Database

I can do this in advance. It doesn’t affect my current environment and doesn’t cause any downtime.

I will move the database into a new Oracle home, so I need to ensure the database configuration files are either outside the Oracle home or move them to the new Oracle home.

  1. I verify that my SP file and password file are stored in ASM – or at least outside the Oracle home:
    [oracle@node1]$ export ORACLE_HOME=$OLD_ORACLE_HOME
[oracle@node1]$ srvctl config database -db $ORACLE_UNQNAME | grep file
    • If the files are stored in the dbs folder, I copy them to new Oracle home.
  2. I copy tnsnames.ora and sqlnet.ora to the new Oracle home:
    [oracle@node1]$ cp $OLD_ORACLE_HOME/network/admin/sqlnet.ora $NEW_ORACLE_HOME/network/admin
[oracle@node1]$ cp $OLD_ORACLE_HOME/network/admin/tnsnames.ora $NEW_ORACLE_HOME/network/admin
  3. I take care of any other configuration files in the Oracle home.
  4. I modify the database so it starts in the new Oracle home on the next restart.
    [oracle@node1]$ srvctl modify database -d $ORACLE_UNQNAME -o $NEW_ORACLE_HOME

4. Switch to the New GI and Database Homes

Now, I can complete the patching process by switching to the new Oracle homes.

  1. I connect as root and start the switch:

    [root@node1]$ export ORACLE_HOME=/u01/app/19.25.0/grid
[root@node1]$ $ORACLE_HOME/rdbms/install/rootadd_rdbms.sh
[root@node1]$ $ORACLE_HOME/crs/install/roothas.sh -prepatch -dstcrshome $ORACLE_HOME

  2. Downtime starts now!

  3. Then, I complete the switch.

    • This step stops the entire GI stack, including resources it manages (databases, listener, etc.).
    • Everything is restarted in the new Oracle homes.
    [root@node1]$ $ORACLE_HOME/crs/install/roothas.sh -postpatch -dstcrshome $ORACLE_HOME

  4. Downtime ends now. Users may connect to the database.

  5. As grid, I update the inventory, so the new GI home is registered as the active one:

    [grid@node1]$ export OLD_ORACLE_HOME=/u01/app/19.24.0/grid
[grid@node1]$ export NEW_ORACLE_HOME=/u01/app/19.25.0/grid
[grid@node1]$ $NEW_ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$NEW_ORACLE_HOME CRS=TRUE
[grid@node1]$ $OLD_ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$OLD_ORACLE_HOME CRS=FALSE

  6. I update any profiles (e.g., .bash_profile) and other scripts referring to the GI home.

  7. As oracle, I update any profiles (e.g., .bash_profile) and other scripts referring to the database home.

5. Complete Patching

  1. I complete patching of the database by running Datapatch (ensure the environment is set correctly):
    [oracle@node1]$ env | grep ORA
[oracle@node1]$ $ORACLE_HOME/OPatch/datapatch

Most likely, there are other changes that you need to make in your own environment:

  • Update Enterprise Manager registration
  • Upgrade RMAN catalog
  • Update other scripts
  • Update /etc/oratab

That’s it! I have now patched my Oracle Restart deployment.

Happy Patching!

Appendix

Deinstall

In the future, I should remove the old Oracle homes. I use the deinstall tool in the respective Oracle homes.

I would recommend waiting a week or two until I’m confident the new Release Updates are fine.

CRS-0245: User doesn’t have enough privilege to perform the operation

  • If you get the following error:
    [oracle@node1]$ srvctl modify database -d $ORACLE_UNQNAME -o $NEW_ORACLE_HOME
PRCD-1163 : Failed to modify database DB19
PRCR-1071 : Failed to register or update resource ora.db19.db
CRS-0245:  User doesn't have enough privilege to perform the operation
  • Be sure to include patch 29326865 in GI and database home.
  • Run the srvctl modify database command as grid instead.
  • Be sure that the Oracle user is still set to oracle after running the command as grid:
    [oracle@node1]$ srvctl config database -db $ORACLE_UNQNAME | grep user

Rollback

If you need to roll back, you more or less reverse the process. The switch home method works to a newer and lower patch level.

OCW Release Update

Thanks to Jan for commenting on the blog post. The initial version didn’t include the OCW Release Update into the database home, which is needed when the database is managed by Grid Infrastructure in any way.

Incorrect Information in ocr.loc

In the ocr.loc file for Oracle Restart, only the local_only property is used. All other properties can be ignored (like ocrconfig_loc).

In Oracle Database 23ai, the file will be cleaner in Oracle Restart. But for Oracle Database 19c there will be these superfluous properties.

Further Reading

Other Blog Posts in This Series

Published

23 thoughts on “How to Patch Oracle Restart 19c and Oracle Database Using Out-Of-Place Switch Home”

  1. Hi,

    thank you for for your blog and all of these very helpful writeups for out-of-place patching.

    Regarding this article today I have one question: Why do you use the RDBMS-patch instead of the GI-patch for the RDBMS ORACLE_HOME? Maybe I missed something changing, but in the past the OCW patch was never included in the RDBMS-patch and only available in the GI-patch, but it’s imperative to also install this patch in the RDBMS ORACLE_HOME if the database instance is managed with Grid Infrastructure. Is this patch included in the RDBMS-patch, now?

    Another, completely independent, thing: Do you plan on doing articles for out-of-place patching of RDBMS and/or Grid Infrastructure on Windows?

    Regards,

    Jan

    Like

    Reply

    1. Hi Jan,

      Thanks for the feedback.

      You’re right about the OCW patch in the database home. This is an oversight coming from too much copy/pasting. It’s good that you caught it, when I failed to do that in my review. I’ll get the post updated soon.

      Regarding the Windows blog posts. It’s a little hard for me to do much on Windows because I don’t have a proper license for it. Most procedures are, however, identical when you take the platform differences into consideration.

      If there’s something specific for Windows you’re looking for – please let me know.

      Thanks,
      Daniel

      Like

      Reply

  2. I, myself, currently don’t have any windows servers I’m working on. But a colleague of mine does, and when we discussed the option to do out-of-place patching, we stumbled across the windows services that need to be handled.

    We think that we have the relevant points figured out (removing the service and recreating it using ORADIM from the new home), but sometimes it’s nice to have confirmation.

    Like

    Reply

    1. Hi Jan,

      Oh yeah, the service will for sure cause problems when you patch Grid Infrastructure and Database at the same time. I don’t see how we can avoid two restarts here.
      Without Grid Infrastructure – if you just want to patch the database out-of-place, that’s super easy with the coming version of AutoUpgrade. One command can do everything – download patches, install oracle home and patch, including registering the service with the correct Oracle home. If your colleague would be interested in trying that, I can set him up with a test version of AutoUpgrade. We hope to release the functionality within a month or two.

      Regards,
      Daniel

      Like

      Reply

  3. Thanks for the offer. I just asked and the next update will be in a few (more than 2 or 3) months. So she said she’ll keep a look out for the new version then.

    Regards, Jan

    Like

    Reply

  4. Hi Daniel,
    I will try to apply patches to my databases, but in production databases we have Data Guard. What is the procedure on Data Guard and Oracle Restart servers?

    Like

    Reply

    1. Hi,
      I have it on my to-do list to produce that blog post as well, but I need more time :)

      I haven’t tested this in my lab, so there might be something that needs to be adjusted. But:
      * Perform task 1, 2 and 3 on all servers
      * Perform task 4 on standby (no datapatch)
      * Perform task 4 on primary (execute datapatch)

      You can also do standby-first patch apply if you want. That could be yet another blog post.

      I hope that helps.

      Regards,
      Daniel

      Like

      Reply

    1. Hi Arti,
      Thanks for reaching out to me. It’s a good question, and normally you would run root.sh for a new installation. However, since we are performing out-of-place patching and switches to a new home, you don’t run root.sh. Instead the other commands (roothas) does the required things.
      Regards,
      Daniel

      Like

      Reply

  7. There is a issue with autoupgrade and out of place upgrade. The issue is that autoupgrade wants to create the target directory instead of using the one provided. It will result in an autoupgrade Info [Exception: COM114. This error comes because the space check is done 1 level up the directory tree. Example: see case SR 3-40203323971.

    Like

    Reply

    1. Hi,

      Thanks for getting in touch with me.

      I understand your use case and I’ll ask a developer to look at it. Once we have a fix, would you be willing to give a test run before we release it officially? If so, I can provide you with a pre-release of AutoUpgrade when it’s ready. If so, please send me an e-mail on daniel.overby.hansen (a) oracle.com.

      Thanks,
      Daniel

      Like

      Reply

  8. Hi Daniel,

    Thanks for this interesting post.

    As this is an Oracle restart, why do you specify the parameter oracle.install.crs.config.clusterName=$CLUSTER_NAME

    in the gridSetup.sh command of the section “prepare a new GI home” ?

    Because it is not a RAC cluster, this parameter is empty in the rsp file of the initial Oracle restart installation.

    Best regards,

    els

    Like

    Reply

    1. Hi Els,

      Probably too much copy/pasting. I started that blog post series with Grid Infrastructure patching and I might have taken too much from there.

      Thanks for letting me know!
      Daniel

      Like

      Reply

      1. Thanks Daniel!

        “copy/paste” can happen, I understand, but with your confirmation, I am sure I can remove this parameter for an Oracle restart.

        See you soon at the “upgrade and migrate” workshop in Brussels :-)

        Like

        Reply

  9. Hi Daniel, thanks for all the great information you share! I’ve got a question: I see that, while preparing the new GI home, you applied the 19.25 Database RU (36912597). I’m trying to do the same out-of-place patching as you but using the latest release (19.27). The README of patch 37641958 (GI Release Update 19.27.0.0.250415), specifically table 1-2, states that the Database RU (37642901 in this case) is applicable for “Only Oracle home for non-Oracle RAC setup. Both Oracle home and Grid home for Oracle RAC setup”. So, just out of curiosity: is it really necessary to apply that patch on the GI home, given that this is not a RAC setup? Thanks!

    Like

    Reply

    1. Hi Hiroshi,

      Thanks for the positive feedback. Much appreciated.

      As you correctly refer from the README, applying the Database RU to the GI home is required only for RAC. At least, that is what the README tells.

      I applied the Database RU to the GI home – even in this case with Oracle Restart – because that’s how I’ve always done it. I’ve mostly done RAC patching where it is required, and to be frank, I wasn’t aware of this difference.

      But it doesn’t “hurt” to apply the Database RU, although not needed for Oracle Restart.

      I’ll try to find some details about that difference.

      I hope that helps,
      Daniel

      Like

      Reply

      1. Hi Daniel! Yes, I did apply the patches, including the DBRU to the Oracle Restart home, there were no issues in the process, and none afterwards (so far). If you do find some info on the difference, please let us know.

        Thanks!

        Like

        Reply

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.