How to Roll Back Oracle Grid Infrastructure 19c Using SwitchGridHome

Grid Infrastructure, Patching 5 Minutes

Let me show you how I roll back a patch from Oracle Grid Infrastructure 19c (GI) using the out-of-place method and the -switchGridHome parameter.

My demo system:

  • Is a 2-node RAC (nodes copenhagen1 and copenhagen2).
  • Runs Oracle Linux.
  • Was patched from 19.17.0 to 19.19.0. I patched both GI and database. Now I want GI back on 19.17.0.

I only roll back the GI home. See the appendix for a few thoughts on rolling back the database as well.

This method works if you applied the patch out-of-place – regardless of whether you used the OPatchAuto or SwitchGridHome method.

Preparation

  • I use the term old Oracle Home for the original, lower patch level Oracle Home.

    • It is my 19.17.0 Oracle Home
    • It is stored in /u01/app/19.0.0.0/grid
    • I refer to this home using the environment variable OLD_ORACLE_HOME
    • This is the Oracle Home that I want to roll back to

  • I use the term new Oracle Home for the higher patch level Oracle Home.

    • It is my 19.19.0 Oracle Home
    • It is stored in /u01/app/19.19.0/grid
    • I refer to this home using the environment variable NEW_ORACLE_HOME
    • This is the Oracle Home that I want to roll back from

Both GI homes are present in the system already.

How to Roll Back Oracle Grid Infrastructure 19c

1. Sanity Checks

I execute the following checks on both nodes, copenhagen1 and copenhagen2. I show the commands for one node only.

  • I verify that the active GI home is the new GI home:

    [grid@copenhagen1]$ export ORACLE_HOME=$NEW_GRID_HOME
[grid@copenhagen1]$ $ORACLE_HOME/srvm/admin/getcrshome

  • I verify that the cluster upgrade state is NORMAL:

    [grid@copenhagen1]$ $ORACLE_HOME/bin/crsctl query crs activeversion -f

  • I verify all CRS services are online:

    [grid@copenhagen1]$ $ORACLE_HOME/bin/crsctl check cluster

  • I verify that the cluster patch level is 19.19.0 – the new patch level:

    [grid@copenhagen1]$ $ORACLE_HOME/bin/crsctl query crs releasepatch

2. Cluster Verification Utility

  • I use Cluster Verification Utility (CVU) to verify that my cluster meets all prerequisites for a patch/rollback. I do this on one node only:
    [grid@copenhagen1]$ $CVU_HOME/bin/cluvfy stage -pre patch
    • You can find CVU in the GI home, but I recommend always getting the latest version from My Oracle Support.

3. Roll Back Node 1

The GI stack (including database, listener, etc.) needs to restart on each instance. But I do the rollback in a rolling manner, so the database stays up all the time.

  • I drain connections from the first node, copenhagen1.

  • I unlock the old GI home, root:

    [root@copenhagen1]$ export OLD_GRID_HOME=/u01/app/19.0.0.0/grid
[root@copenhagen1]$ cd $OLD_GRID_HOME/crs/install
[root@copenhagen1]$ ./rootcrs.sh -unlock -crshome $OLD_GRID_HOME
    • This is required because the next step (gridSetup.sh) runs as grid and must have access to the GI home.
    • Later on, when I run root.sh, the script will lock the GI home.

  • I switch to old GI home as grid:

    [grid@copenhagen1]$ export OLD_GRID_HOME=/u01/app/19.0.0.0/grid
[grid@copenhagen1]$ export ORACLE_HOME=$OLD_GRID_HOME
[grid@copenhagen1]$ export CURRENT_NODE=$(hostname)
[grid@copenhagen1]$ $ORACLE_HOME/gridSetup.sh \
   -silent -switchGridHome \
   oracle.install.option=CRS_SWONLY \
   ORACLE_HOME=$ORACLE_HOME \
   oracle.install.crs.config.clusterNodes=$CURRENT_NODE \
   oracle.install.crs.rootconfig.executeRootScript=false

  • I complete the switch by running root.sh as root:

    [root@copenhagen1]$ export OLD_GRID_HOME=/u01/app/19.0.0.0/grid
[root@copenhagen1]$ $OLD_GRID_HOME/root.sh
    • This step restarts the entire GI stack, including resources it manages (databases, listener, etc.). This means downtime on this node only. The remaining nodes stay up.
    • In that period, GI marks the services as OFFLINE so users can connect to other nodes.
    • If my database listener runs out of the Grid Home, GI will move it to the new Grid Home, including copying listener.ora.
    • In the end, GI restarts the resources (databases and the like).

  • I update any profiles (e.g., .bashrc) and other scripts referring to the GI home.

  • I verify that the active GI home is the new GI home:

    [grid@copenhagen1]$ $OLD_ORACLE_HOME/srvm/admin/getcrshome

  • I verify that the cluster upgrade state is ROLLING PATCH:

    [grid@copenhagen1]$ $OLD_ORACLE_HOME/bin/crsctl query crs activeversion -f

  • I verify all CRS services are online:

    [grid@copenhagen1]$ $OLD_ORACLE_HOME/bin/crsctl check cluster

  • I verify all resources are online:

    [grid@copenhagen1]$ $OLD_ORACLE_HOME/bin/crsctl stat resource -t

  • I verify that the GI patch level is 19.17.0 – the old patch level:

    [grid@copenhagen1]$ $OLD_ORACLE_HOME/bin/crsctl query crs releasepatch

4. Roll Back Node 2

  • I roll back the second node, copenhagen2, using the same process as the first node, copenhagen1.
    • I double-check that the CURRENT_NODE environment variable gets updated to copenhagen2.
    • When I use crsctl query crs activeversion -f to check the cluster upgrade state, it will now be back in NORMAL mode, because copenhagen2 is the last node in the cluster.

5. Cluster Verification Utility

  • I use Cluster Verification Utility (CVU) again. Now I perform a post-rollback check. I do this on one node only:
    [grid@copenhagen1]$ $CVU_HOME/bin/cluvfy stage -post patch

That’s it!

My cluster is now operating at the previous patch level.

Appendix

SwitchGridHome Does Not Have Dedicated Rollback Functionality

OPatchAuto has dedicated rollback functionality that will revert the previous patch operation. Similar functionality does not exist when you use the SwitchGridHome method.

This is described in Steps for Minimal Downtime Grid Infrastructure Out of Place ( OOP ) Patching using gridSetup.sh (Doc ID 2662762.1). To rollback, simply switch back to the previous GI home using the same method as for the patch.

There is no real rollback option as this is a switch from OLD_HOME to NEW_HOME To return to the old version you need to recreate another new home and switch to that.

Should I Roll Back the Database as Well?

This post describes rolling back the GI home only. Usually, I recommend keeping the database and GI patch level in sync. If I roll back GI, should I also roll back the database?

The short answer is no!

Keeping the GI and database patch in sync is a good idea. But when you need to roll back, you are in a contingency. Only roll back the component that gives you problems. Then, you will be out of sync for a period of time until you can get a one-off patch or move to the next Release Update. Being in this state for a shorter period is perfectly fine – and supported.

Other Blog Posts in This Series

Published

11 thoughts on “How to Roll Back Oracle Grid Infrastructure 19c Using SwitchGridHome”

  1. Hi Daniel,

    thank you for all the work you put into the blog explaining the GI out of place patching for RAC. But I wonder how to do the same with the database in RAC? Is that possible with out of place and RAC rolling as well? I can only imagine out of place patching for the database to be a clusterwide operation and so with downtime. Is that correct? Are there any blogs from you or mike which cover that? I couldn’t find any :-(.

    Best regards,

    Robert

    Like

    Reply

    1. Hi Robert,

      Thanks for the nice feedback – much appreciated.

      Sure, you can do the same with the database as well. You can use out-of-place patching for GI and the database and do it all in one operation. When the GI stack restarts on one instance, it will restart the database in the new Oracle home as well.

      Check out these two blog posts:

      How to Apply Patches Out-of-place to Oracle Grid Infrastructure and Oracle Data Guard Using Standby-First

      Patching Oracle Grid Infrastructure And Oracle Data Guard

      It’s about Data Guard, but you can use the same approach for single instance databases as well. You srvctl modify database to tell GI to start the DB in the new Oracle home when it restarts next time. When all instances are running on the new Oracle home, you can run datapatch.

      Let me know how it turns out!

      Regards,
      Daniel

      Like

      Reply

  2. Hi Daniel,

    thank you for all the work you put into the blog and explaining the GI out of place patching for RAC. But I wonder if the same is possible with the database as well? I know I can do out of place patching and the home switch basically with

    srvctl stop database -d DB

    srvctl modify database -d DB -oraclehome /path/to/new/home

    srvctl start database -d DB

    But this is a clusterwide operation and so causes downtime. Is database out of place patching possing on RAC in a rolling manner? How? Any blogs from Mike or your on that matter? I can’t find any … :-(.

    Many thanks,

    Robert

    Like

    Reply

  3. Hi Daniel,

    many thanks for that quick response. The core basically is, that I can modify the database resource home while all instances run on the old home. When restarting the instances one at a time they use the new home. Once all instances are restarted the home switch for the database is done. Is that correct?

    And is there something similar to “crsctl query crs activeversion -f” for the database to verify its patching state and version in the whole cluster?

    Many thanks,

    Robert

    Like

    Reply

    1. Hi Robert,

      Yes, that’s correct. You tell GI to start the database in the new home on next restart. You can do that rolling, so database instances will gradually move to the new home. Then, when all instances are on the new home, you run datapatch one time on any of the nodes.

      There is no similar function in the database that you can use. Or, wait…. I see there there are two interesting functions in DBMS_QOPATCH. I’ve never used them, so, if you try them out, let me know how it turns out :)
      https://docs.oracle.com/en/database/oracle/oracle-database/19/arpls/DBMS_QOPATCH.html#GUID-D754EFEE-1AA4-4FC9-861A-82CFE709678B
      https://docs.oracle.com/en/database/oracle/oracle-database/19/arpls/DBMS_QOPATCH.html#GUID-4DDB00C1-7603-4F82-BEA7-A584F5D22CB5

      Regards,
      Daniel

      Like

      Reply

  4. Hi Daniel,

    today I had the time to test that. But one after the other, as there are many points:

    1st) DBMS_QOPATCH.OPATCH_COMPARE_NODES works for that. With its output and a little grep’ing I can get a boolean state for PATCHES_IN_SYNC. This is all I need. Very well.

    PS: The output is kind of weird and here and there a little broken. I will not open an SR just for that … maybe you can forward that.

    —DBMS_QOPATCH.OPATCH_COMPARE_NODES Output —Current node is marked as target Database running instance <=== missing SID here?Comparing with sol08p0002(reference database) and instance PLAY12

    Patch Not found in Target Database

    Patch(es) Not found in <=== weird line break?Reference Database

    Patch(es) which are common Patch 35858371 Patch 35787077 Patch 35739076 Patch 35648110 Patch 35638318 Patch 35403248 Patch 34774667 Patch 34304965 Patch 35655527<=== weird empty line?Patch 35643107 <=== missing spaces in front——————————————————————–

    No big issue, works for me even with those weird issues.Thank you.Robert

    Like

    Reply

  5. And my last comment for today :-).

    3rd) So, the pdbs. Just for reference, I am patching from 19.21 to 19.23 with some patches on top.

    All were open unrestricted on all instances before I started. Same for PDB$SEED but in read only mode.

    After first instance has been switched to the new home and restarted, that was again the case for ALL pdbs on all instances. Fine.

    After the second and last instance has been switched to the new home and restarted, things start to get weird.

    on first instance, all pdbs were unchanged: the custom pdbs were all in read write unrestricted mode, PDB$SEED in read only unrestricted mode.

    on second instance, PDB$SEED was opened read only unrestricted and all custom pdbs were closed. I can open the custom pdbs here only in restricted mode and with errors.

    Where to run datapatch now? Do I have to go back to my first instance and run datapatch there with all PDBs in the correct state? Or should I open the custom pdbs in restricted mode on the second instance, run datapatch there and close + reopen them afterwards? Any recommendations here?

    Thank you very much,Robert

    Like

    Reply

  7. Hi Daniel,

    just a short question concerning the rollback and the mentioned Doc ID 2662762.1.

    To return to the old version you need to recreate another new home and switch to that.

    When I patch OOP I still have my OLD_HOME. But my guess is that this record is related to a former statement in this doc:

    To clean up the old home post switch use 1346305.1

    Right?

    By the way: I had similar problems as Robert with pdbs in restricted mode after OOP patching from 19.22.0 to 19.26.0 using an golden image and it took one plus 4 spinoff SRs to fix this. And I had to blame myself as I missed an old and unused Java RU on this system ;-)

    Best regards

    Axel D.

    Like

    Reply

    1. Hi Axel,

      You don’t have to create a new Oracle home if you want to roll back. You can roll back to the previous Oracle home – if you haven’t deleted it yet.

      You can leave a comment in My Oracle Support on the note, asking them to change the text. It has much greater value when such a request comes from a customer.

      Regards,
      Daniel

      Like

      Reply

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.