How to Apply Patches Out-of-place to Oracle Grid Infrastructure and Oracle Data Guard Using Standby-First

I strongly recommend that you always patch out-of-place. Here’s an example of how to do it on Oracle Grid Infrastructure (GI) and Oracle Data Guard using Standby-First Patch Apply.

Standby-First Patch Apply allows you to minimize downtime to the time it takes to perform a Data Guard switchover. Further, it allows you to test the apply mechanism on the standby database by temporarily converting it into a snapshot standby database.

The scenario:

  • Oracle Grid Infrastructure 19c and Oracle Database 19c
  • Patching from Release Update 19.17.0 to 19.19.0
  • Vertical patching – GI and database at the same time
  • Data Guard setup with two RAC databases
    • Cluster 1: copenhagen1 and copenhagen2
    • Cluster 2: aarhus1 and aarhus2
    • DB_NAME: CDB1
    • DB_UNIQUE_NAME: CDB1_COPENHAGEN and CDB1_AARHUS
  • Using Data Guard broker
  • Patching GI using SwitchGridHome method

Let’s get started!

Step 1: Prepare

I can make the preparations without interrupting the database.

Step 2: Restart Standby in New Oracle Homes

Now, I can move the standby database to the new GI and database homes.

  • On the standby hosts, aarhus1 and aarhus2, I first move the database configuration files from the old database home to the new one.

  • I change the database configuration in GI. Next time the database restarts, it will be in the new Oracle Home:

    [oracle@aarhus1]$ $OLD_ORACLE_HOME/bin/srvctl modify database \
       -db $ORACLE_UNQNAME \
       -oraclehome $NEW_ORACLE_HOME
    
  • I switch to the new GI on all standby hosts, aarhus1 and aarhus2, by executing step 2 (Switch to the new Grid Home) of the SwitchGridHome method.

    • It involves running gridSetup.sh ... -switchGridHome and root.sh.
    • You can perform the switch in a rolling manner or all at once.
    • The switch restarts the standby database instance. The standby database instance restarts in the new Oracle Home.
    • If the profile of grid (like .bashrc) sets the ORACLE_HOME environment variable, I ensure to update it.
  • If I had multiple standby databases, I would process all standby databases in this step.

Step 3: Test Standby Database

This is an optional step, but I recommend that you do it.

  • I convert the standby database (CDB1_AARHUS) to a snapshot standby database:
    DGMGRL> convert database CDB1_AARHUS to snapshot standby;
    
  • I test Datapatch on the standby database. It is important that I run the command on the standby database:
    [oracle@aarhus1]$ $ORACLE_HOME/OPatch/datapatch -verbose
    
  • I can also test my application on the standby database.
  • At the end of my testing, I revert the standby database to a physical standby database. The database automatically reverts all the changes made during testing:
    DGMGRL> convert database CDB1_AARHUS to physical standby;
    

Step 4: Switchover

I can perform the previous steps without interrupting my users. This step requires a maintenance window because I am doing a Data Guard switchover.

  • I check that my standby database is ready to become primary. Then, I start a Data Guard switchover:
    DGMGRL> connect sys/<password> as sysdba
    DGMGRL> validate database CDB1_AARHUS;
    DGMGRL> switchover to CDB1_AARHUS;
    

A switchover does not have to mean downtime.

If my application is configured properly, the users will experience a brownout; a short hang, while the connections switch to the new primary database.

Step 5: Restart New Standby in New Oracle Homes

Now, the primary database runs on aarhus1 and aarhus2. Next, I can move the new standby hosts, copenhagen1 and copenhagen2, to the new GI and database homes.

  • I repeat step 2 (Restart Standby In New Oracle Homes) but this time for the new standby hosts, copenhagen1 and copenhagen2.

Step 6: Complete Patching

Now, both databases in my Data Guard configuration run out of the new Oracle Homes.

Only proceed with this step once all databases run out of the new Oracle Home.

I need to run this step as fast as possible after I have completed the previous step.

  • I complete the patching by running Datapatch on the primary database (CDB1_AARHUS). I add the recomp_threshold parameter to ensure Datapatch recompiles all objects that the patching invalidated:

    [orale@aarhus1]$ $ORACLE_HOME/OPatch/datapatch \
       -verbose \
       -recomp_threshold 10000
    
    • I only need to run Datapatch one time. On the primary database and only on one of the instances.
  • I can run Datapatch while users are connected to my database.

  • Optionally, I can switch back to the original primary database on copenhagen1 and copenhagen2, if I prefer to run it there.

That’s it. Happy patching!

Appendix

Further Reading

Other Blog Posts in This Series

10 thoughts on “How to Apply Patches Out-of-place to Oracle Grid Infrastructure and Oracle Data Guard Using Standby-First

  1. Hi, wondering if Oracle Restart or single-node Oracle GI cluster was used in this scenario? MOS Doc ID 2853839.1 says “-switchGridHome” is not applicable to Oracle Restart.

    Like

  2. Daniel, thanks for the info.
    I did a new install of Oracle19c Data Guard instead of upgrading my Oracle12c Data Guard, I am using Protection Mode=MaxPerformance in 12c, but in 19c Protection Mode=MaxPerformance I see the message “Fast-Start Failover: Enabled in Potential Data Loss Mode”, is this a BUG?

    regards,
    jorge

    Like

  3. Hi Jorge,
    I don’t know. For such questions on Data Guard, I suggest that you reach out to the Data Guard Product Manager, Ludovico Caldera. You can find him on Twitter as @ludodba.
    Regards,
    Daniel

    Like

  4. Hej Daniel,
    Thanks for your post.

    But I still have a question regarding the use of SwitchGridHome method in Single Instance DB managed by GI. You mentioned that Oracle Restart is NOT supported with SwitchGridHome method . So does it mean, we need to disable Oracle Restart in case we want to use SwitchGridHome method? As of now we follow below steps for GoldImage based GI Patching ! Can we replace them with SwitchGridHome ?

    After extracting GoldImage of Target Home – For Example: 19.19 GoldImage on Target Server, we follow below steps.

    1. Execute gridSetup.sh (From Target Home)
    2. Execute root.sh (From Target Home)
    3. roothas.sh -prepatch -dstcrshome $TARGET_GRID_HOME_RU (From Target Home)
    4. roothas.sh -postpatch -dstcrshome $TARGET_GRID_HOME_RU (From Target Home)
    5. ./oui/bin/runInstaller -updateNodeList ORACLE_HOME=$TARGET_GRID_HOME_RU CRS=”true” (From Target Home)
    6. ./oui/bin/runInstaller -updateNodeList ORACLE_HOME=$SOURCE_GRID_HOME CRS=”false” (From Source Home)
    7. srvctl modify listener -l LISTENER -o $TARGET_GRID_HOME_RU
    8. Stop/Start Listener

    Tak på forhånd

    Like

  5. Hi,

    The procedure I describe work for Grid Infrastructure only – a RAC managed system. If you have a single instance database, then the procedure is not supported. You have to use another approach:
    https://docs.oracle.com/en/database/oracle/oracle-database/19/ladbi/switch-gi-home-patching.html#GUID-27EBDA86-24B2-407B-A4AD-C0542B472F90

    I have it on my list of blog posts to write. But that list keeps growing and growing and ….. 🙂

    Regards,
    Daniel

    Like

  6. in step 3 of How to Apply Patches Out-of-place to Oracle Grid Infrastructure and Oracle Data Guard Using Standby-First, it mentions that the step is optional

    is that for all three components of that step or just for the testing… if it is optional, then when does datapatch acutally get run (is it at the end of the full process?)

    Like

  7. Hi Ray,

    It is all of step 3 that is optional. Since this is for testing only, you can run Datapatch even though the primary database hasn’t been patched yet. With Datapatch you can test the SQL part of the patch apply on the standby database only. When you’re done testing all changes are reverted when you convert back to physical standby database (as if Datapatch hasn’t been executed at all).
    Then at the end of step 6 – when all databases in the Data Guard configuration is on the new Oracle Home, you run Datapatch on the primary databases. The SQL patches then propagates to the standby databases via redo.

    Regards,
    Daniel

    Like

  8. Hi Daniel, Im currently busy with the next RU patch cycle. Of course Im using OOP method (even though I prefer opatchauto up to now 😉) and Standby-first if possible.

    For one topic I still did not find a corresponding note within the documentation: after applying the RU on the primary (and maybe additional One-Offs, e.g. the DP-bundle) I usually enable the optimizer fixes on the primary. For the db related thing its fine due to redo but what about the param _fix_control? It has to be set to the spfile of the standby accordingly, right?

    Thanks & regards

    Axel D.

    Like

  9. Hi Axel,

    When you enable optimizer fixes on primary database, the changes do not apply to the standby database. To use DBMS_OPTIM_BUNDLE on the standby database, you must open it (read-only – no redo apply if you don’t have Active Data Guard) or switchover to the standby database to use the package.
    An alternative is to find the _fix_control setting in the SPFile on the primary and set the same parameter on the standby database using ALTER SYSTEM.

    Regards,
    Daniel

    Like

Leave a comment