How to Patch Oracle Net Listener

When running Oracle Database on a server, the Oracle Net Listener is a vital component.

Let’s discuss how you can patch it.

The Basics

Normally, on a server, you have:

One listener
One or many database instances

That one listener handles new connections to all database instances. You patch the listener simply by starting it in the new Oracle home.

You can have multiple listeners and use remote listeners, but if you do that, you’re smart and probably won’t need the advice in this blog post.

How to Patch the Listener

Copy listener.ora from the old Oracle home to the new Oracle home.
```
export OLD_ORACLE_HOME=/u01/app/oracle/product/19.26.0/dbhome_1
export NEW_ORACLE_HOME=/u01/app/oracle/product/19.27.0/dbhome_1
cp $OLD_ORACLE_HOME/network/admin/listener.ora $NEW_ORACLE_HOME/network/admin
```
- You can avoid this step by redirecting the location of your network files using the TNS_ADMIN environment variable.
- If you have a completely default installation with no listener configuration, you don’t have a listener.ora file and can skip this step.
- In an advanced configuration you might have references to the listener Oracle home. Those you must update. But don’t confuse the listener Oracle home, with the Oracle homes of the databases with static registration. Those you update, when you patch the matching database.
Restart the listener in the new Oracle home:
```
export ORACLE_HOME=$OLD_ORACLE_HOME
$OLD_ORACLE_HOME/bin/lsnrctl stop
export ORACLE_HOME=$NEW_ORACLE_HOME
$NEW_ORACLE_HOME/bin/lsnrctl start 
```
- If you use Oracle Grid Infrastructure or Oracle Restart, they manage the listener. You automatically restart, and thus patch, the listener when you patch those components.
If you use the dbstart script to start the listener, update it with the new Oracle home.
- On Windows, you must delete the matching Windows service when you stop the listener. The lsnrctl start command should create a new listener. Be sure to set it to start automatically.

The Outage

When you restart the listener, there’s a short period where there’s no listener. It takes a second or so for the new listener to start. In that period, any new connection attempt fails:
```
ORA-12541: TNS:no listener
```
This affects new connections only. Existing connections continue to work unaffected.
As soon as the listener starts, it’ll read listener.ora and go through the static listener registrations defined by the SID_LIST. Connections to those databases are ready immediately.
For optimal flexibility, however, it’s better to use dynamic listener registration, where the database instance automatically registers with the listener. When you restart the listener, it doesn’t know of all the dynamic registrations and will have to wait until the database registers (up to 60 seconds).
- You can force a database to register with the listener:
```
SQL> alter system register;
```
- It’s a good idea to run through all of your databases after the listener restart and force a listener registration with the above command.
- Until the dynamic registration completes, any new connection to the database will fail:
```
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
```
You can hide the short outage from the client by using RETRY_COUNT and RETRY_DELAY in your connect strings. Allow the client to retry for a short period before throwing the above errors.
You can avoid the outage completely by using multiple listeners and patching them separately. Your connect strings should have an ADDRESS_LIST containing both listeners.

Do I Need to Patch the Listener and Database At the Same Time?

No, that’s up to you. If you want just one interruption, then do it all at once. Otherwise, you can do it in separate maintenance windows.

What About AutoUpgrade?

At the moment, AutoUpgrade doesn’t patch the listener. But it’s on the backlog.

Happy patching!

AutoUpgrade New Features: Better Automation To Patch Oracle Database on Windows

Running Oracle Database on Microsoft Windows is slightly different from running it on other platforms. So, of course, patching Oracle Database is also slightly different.

The Oracle Database runs as a Windows service. AutoUpgrade must re-create the service when you perform out-of-place patching so the service starts oracle.exe from the new Oracle home.

To recreate the service, you must specify the credentials of the user who runs the service. Windows allows you to store the credentials in a special file; AutoUpgrade can use that when it recreates the service.

For security purposes, AutoUpgrades deletes the credential file when it is no longer needed. For automation, however, that’s impractical because you would need to recreate the credential file every time you patch or upgrade.

AutoUpgrade now allows you to keep the file and reuse it. To do so, use the config file parameter delete_credential_file.

How To Patch Oracle Database on Windows

Get the latest version of AutoUpgrade:

wget https://download.oracle.com/otn-pub/otn_software/autoupgrade.jar

Create an AutoUpgrade config file:

global.keystore=c:\oracle\autoupgrade\keystore
patch1.source_home=c:\oracle\product\dbhome_19_26_0
patch1.target_home=c:\oracle\product\dbhome_19_27_0
patch1.sid=DB19
patch1.folder=c:\oracle\patches
patch1.patch=RECOMMENDED
patch1.wincredential=c:\oracle\autoupgrade\credential
patch1.delete_credential_file=false

wincredential is the location where AutoUpgrade stores the credential file.
Set delete_credential_file to false to keep the credential file so you can re-use it.

Load the credentials for the user running the service into a credential file:

java -jar autoupgrade.jar 
     -config ...
     –patch 
     -load_win_credential "DB19"

Start AutoUpgrade in deploy mode:
```
java -jar autoupgrade.jar 
     -config ...
     –patch 
     -mode deploy
```
- AutoUpgrade finds and downloads the right patches for Windows.
- Creates a new Oracle home with the new patches.
- Completes the entire patching process.

That’s it! You’ve patched your Oracle Database on Windows.

Here’s a little demo from our YouTube channel. Be sure to subscribe so you don’t miss out.

Happy patching!

Introduction to Patching Oracle Data Guard

Here’s a blog post series about patching Oracle Data Guard in single instance configuration. For simplicity, I am patching with Oracle AutoUpgrade to automate the process as much as possible.

First, a few ground rules:

Always use out-of-place patching
You must have the same set of patches on primary and standby

The Methods

There are three ways of patching Data Guard:

All At Once

You patch all databases at the same time.
You need an outage until you’ve patched all databases.
You need to do more work during the outage.
You turn off redo transport while you patch.

Standby-first with restart

All the patches you apply must be standby-first installable (see appendix).
You need an outage to stop the primary database and restart it in the target Oracle home.
During the outage, you have to do less work to do compared to all at once and less work overall compared to standby-first with switchover.
The primary database remains the same. It is useful if you have an async configuration with a much more powerful primary database or just prefer to have a primary database at one specific location.

Standby-first with switchover

All the patches you apply must be standby-first installable (see appendix).
You need an outage to perform a switchover. If your application is well-configured, users will just experience it as a brownout (hanging for a short period while the switchover happens).
During the outage, you have little to do, but overall, there are more steps.
After the outage, if you switch over to an Active Data Guard, the workload from the read-only workload has pre-warmed the buffer cache and shared pool.

Summary

All at one	Standby-first with restart	Standby-first with switchover
Works for all patches	Works for most patches	Works for most patches
Bigger interruption	Bigger interruption	Smaller interruption
Downtime is a database restart	Downtime is a database restart	Downtime/brownout is a switchover
Slightly more effort	Least effort	Slightly more effort
Cold database	Cold database	Pre-warmed database if ADG

Here’s a decision tree you can use to find the method that suits you.

What If

RAC

These blog posts focus on single instance configuration.

Conceptually, patching Data Guard with RAC databases is the same; you can’t use the step-by-step guides in this blog post series. Further, AutoUpgrade doesn’t support all methods of patching RAC databases (yet).

I suggest that you take a look at these blog posts instead:

Or even better, use Oracle Fleet Patching and Provisioning.

Oracle Restart

You can use these blog posts if you’re using Oracle Restart. You can even combine patching Oracle Restart and Oracle Database into one operation using standby-first with restart.

We’re Really Sensitive To Downtime?

In these blog posts, I choose the easy way – and that’s using AutoUpgrade. It automates many of the steps for me and has built-in safeguards to ensure things don’t go south.

But this convenience comes at a price: sligthly longer outage. Partly, because AutoUpgrade doesn’t finish a job before all post-upgrade tasks are done (like Datapatch and gathering dictionary stats).

If you’re really concerned about downtime, you might be better off with your own automation, where you can open the database for business as quickly as possible while you run Datapatch and other post-patching activities in the background.

Datapatch

Just a few words about patching Data Guard and Datapatch.

You always run Datapatch on the primary.
You run Datapatch just once, and the changes to the data dictionary propagates to the standby via redo.
You run Datapatch when all databases are running out of the new Oracle home or when redo transport is turned off. The important part is that the standby that applies the Datapatch redo must be on the same patch level as the primary.

Happy patching

Appendix

Standby-First Installable

You can only perform standby-first patch apply if all the patches are marked as standby-first installable.

Standby-first patch apply is when you patch the standby database first, and you don’t disable redo transport/apply.

You can only use standby-first patch apply if all the patches are classified as standby-first installable. For each of the patches, you must:

Examine the patch readme file.
One of the first lines will tell if this specific patch is standby-first installable. It typically reads: > This patch is Data Guard Standby-First Installable

Release Updates are always standby-first installable, and so are most of the patches for Oracle Database.

In rare cases, you find a non-standby-first installable patch, so you must patch Data Guard using all at once.

How To Patch Oracle Data Guard Using AutoUpgrade And Standby-First Patch Apply With Switchover

Let me show you how I patch my Oracle Data Guard configuration. I make it as easy as possible using Oracle AutoUpgrade. I reduce the interruption by doing standby-first patch apply with a switchover.

My Data Guard configuration consists of two databases:
- SID: SALES
- Databases: SALES_COPENHAGEN and SALES_AARHUS
- Hosts: copenhagen and aarhus
- Primary database: SALES_COPENHAGEN running on copenhagen

Preparations

You should do these preparations in advance of your maintenance window. They don’t interupt operations on your databases.

I download the patches using AutoUpgrade.
- I can download the patches from any computer. It doesn’t have to be one of the database hosts, which typically don’t have internet access.
I verify all patches are standby-first installable and my configuration meets the requirements for standby-first patch apply.

I create a new Oracle home on all hosts.

Create a config file called sales.cfg:

global.global_log_dir=/home/oracle/autoupgrade-patching/sales
patch1.source_home=/u01/app/oracle/product/19.3.0.0/dbhome_1
patch1.target_home=/u01/app/oracle/product/19/dbhome_19_26_0
patch1.sid=SALES
patch1.folder=/home/oracle/autoupgrade-patching/patch
patch1.patch=RECOMMENDED,MRP
patch1.download=no

Start AutoUpgrade in create_home mode:

java -jar autoupgrade.jar -config sales.cfg -patch -mode create_home

AutoUpgrade also runs root.sh if oracle user has sudo privileges. Else, I must manually execute root.sh.

Optionally, but recommended, I run an analysis on the primary database:
```
[oracle@copenhagen] java -jar autoupgrade.jar -config sales.cfg -patch -mode analyze
```
- Check the findings in the summary report.

Patching

Proceed with the following when your maintenance window starts.

Update listener.ora on the standby host (see appendix). I change the ORACLE_HOME parameter in the static listener entry (suffixed _DGMGRL) so it matches my target Oracle home.
I reload the listener:
```
[oracle@aarhus] lsnrctl reload
```

Patch the standby database:

[oracle@aarhus] java -jar autoupgrade.jar -config sales.cfg -mode deploy

I don’t disable redo transport/apply.

Optionally, test the application of patches using a snapshot standby database.
interruption starts!
Switch over to SALES_AARHUS:
```
DGMGRL> switchover to sales_aarhus;
```
- Perform draining in advance according to your practices.
- Depending on how your application is configured, the users will experience this interruption as a brown-out or downtime.
Update listener.ora on the new standby host (copenhagen). I change the ORACLE_HOME parameter in the static listener entry (suffixed _DGMGRL) so it matches my target Oracle home.
I reload the listener:
```
[oracle@copenhagen] lsnrctl reload
```

Patch the new standby database (see appendix):

[oracle@copenhagen] java -jar autoupgrade.jar -config sales.cfg -mode deploy

Verify the Data Guard configuration and ensure the standby database is receiving and applying redo:

DGMGRL> show database SALES_COPENHAGEN;
DGMGRL> show database SALES_AARHUS;
DGMGRL> validate database SALES_COPENHAGEN;
DGMGRL> validate database SALES_AARHUS;

Post-Patching

Connect to the new primary database and execute Datapatch. You do that by calling AutoUpgrade in upgrade mode:
```
[oracle@aarhus] java -jar autoupgrade.jar -config sales.cfg -mode upgrade -clear_recovery_data
```
- Since I’m reusing the same config file, I must add the -clear_recovery_data flag. Otherwise, AutoUpgrade gets a little confused.

Happy Patching!

Appendix

Static Listener Entry

In this blog post, I update the static listener entries required by Data Guard broker (suffixed DGMGRL). My demo environment doesn’t use Oracle Restart or Oracle Grid Infrastructure, so this entry is mandatory.

If you use Oracle Restart or Oracle Grid Infrastructure, such entry is no longer needed.

Avoid Problems on the Primary Database by Testing on a Snapshot Standby

One of the advantages of standby-first patch apply, is that I can test the patches in a production-like environment (the standby) before applying them to the primary. Should I find any issues with the patches, I can stop the process and avoid impacting the primary database.

Here’s an overview of the process.

For demo purposes, my Data Guard configuration consists of two databases:

SID: SALES
Databases: SALES_COPENHAGEN and SALES_AARHUS
Hosts: copenhagen and aarhus
Primary database: SALES_COPENHAGEN running on copenhagen

How To

This procedure starts right after I’ve patched the standby (SALES_AARHUS). It runs out of the target Oracle home, whereas the primary database (SALES_COPENHAGEN) still runs on the source Oracle home.

Convert the patched standby to a snapshot standby:
```
DGMGRL> convert database SALES_AARHUS to snapshot standby;
```
- The database sets a guaranteed restore point before opening the standby in READ WRITE mode.
Test the patch apply by running Datapatch on the standby:
```
[oracle@aarhus] $ORACLE_HOME/OPatch/datapatch
```
- One always runs Datapatch on the primary database and the changes made by the patches goes into redo to the standby.
- But, since I converted to a snapshot standby, it is now opened like a normal database and I can run Datapatch on it.
- If Datapatch completes without problems on the standby, I can be pretty sure it will do so on the primary as well. The standby is after all an exact copy of the primary database.
Optionally, perform additional testing on the standby.
- I can connect any application and perform additional tests.
- I can use SQL Performance Analyzer to check for regressing SQL statements.
- I can make changes to any data in the standby. It is protected by a restore point.
When done, convert the snapshot standby back to a physical standby:
```
DGMGRL> convert database SALES_AARHUS to physical standby;
```
- This implicitly shuts down the standby, flashes back to the restore point and re-opens the database as a physical standby.
- All changes made when it was a snapshot standby, including the Datapatch run, are undone.

Continue the patching procedure on the primary database as described elsewhere.

Is It Safe?

Sometimes, when I suggest using the standby for testing, people are like: Huh! Seriously?

What Happens If I Need to Switch Over or Fail Over?

I can still perform a switchover or a failover. However, they will take a little bit longer.

When I convert to snapshot standby:

Redo transport is still active.
Redo apply is turned off.

So, the standby receives all redo from the primary but doesn’t apply it. Since you normally test for 10-20 minutes, this would be the maximum apply lag. On a well-oiled standby, it shouldn’t take more than a minute or two to catch up.

When performing a switchover or failover on a snapshot standby, you should expect an increase with the time it takes to:

Shut down
Flashback
Apply redo

I’d be surprised if that would be more than 5 minutes. If your RTO doesn’t allow for a longer period:

Get a second standby.
Consider the reduction in risk you get when you test on the standby. Perhaps a short increase in RTO could be allowed after all.

What Happens If Datapatch Fails

If Datapatch fails on my snapshot standby, I should be proud of myself. I just prevented the same problem from hitting production.

I grab all the diagnostic information I need, so I can work with Oracle Support on the issue.
Convert back to physical standby, which will undo the failed Datapatch run.
If I expect to solve the issue quickly, leave the standby running in the target Oracle home. Otherwise, put it back into the source Oracle home.

So, yes, it’s safe to use!

Happy testing

Appendix

How To Patch Oracle Data Guard Using AutoUpgrade For Non-Standby-First Installable Patches

Let me show you how I patch my Oracle Data Guard configuration. I make it easy using Oracle AutoUpgrade. I patch all at once – all databases at the same time – which means a short downtime. I can use this approach for all patches – even those that are not standby-first installable.

I recommend this approach only when you have patches that are not standby-first installable.

My Data Guard configuration consists of two databases:
- SID: SALES
- Databases: SALES_COPENHAGEN and SALES_AARHUS
- Hosts: copenhagen and aarhus
- Primary database: SALES_COPENHAGEN running on copenhagen

Preparations

You should do these preparations in advance of your maintenance window. They don’t interupt operations on your databases.

I download the patches using AutoUpgrade.
- I can download the patches from any computer. It doesn’t have to be one of the database hosts, which typically don’t have internet access.

I create a new Oracle home on all hosts.

Create a config file called sales.cfg:

global.global_log_dir=/home/oracle/autoupgrade-patching/sales
patch1.source_home=/u01/app/oracle/product/19.3.0.0/dbhome_1
patch1.target_home=/u01/app/oracle/product/19/dbhome_19_26_0
patch1.sid=SALES
patch1.folder=/home/oracle/autoupgrade-patching/patch
patch1.patch=RECOMMENDED,MRP
patch1.download=no

Start AutoUpgrade in create_home mode:

java -jar autoupgrade.jar -config sales.cfg -patch -mode create_home

AutoUpgrade also runs root.sh if oracle user has sudo privileges. Else, I must manually execute root.sh.

Optionally, but recommended, I run an analysis on the primary database:
```
[oracle@copenhagen] java -jar autoupgrade.jar -config sales.cfg -patch -mode analyze
```
- Check the findings in the summary report.

Patching

Proceed with the following when your maintenance window starts.

I connect to the primary database using Data Guard broker and disable redo transport from the primary database:
```
DGMGRL> edit database sales_copenhagen set state='TRANSPORT-OFF';
```
I update listener.ora on both hosts (see appendix). I change the ORACLE_HOME parameter in the static listener entry (suffixed _DGMGRL) so it matches my target Oracle home.
I reload the listener on both hosts:
```
lsnrctl reload
```
Downtime starts!
- Perform draining in advance according to your practices.
- Shut down your application.

Patch the primary database:

[oracle@copenhagen] java -jar autoupgrade.jar -config sales.cfg -patch -mode deploy

Simultaneously, I patch the standby database:

[oracle@aarhus] java -jar autoupgrade.jar -config sales.cfg -mode deploy

I update my profile and scripts so they point to the target Oracle home.

When patching completes in both hosts, I re-enable redo transport:

DGMGRL> edit database sales_copenhagen set state='TRANSPORT-ON';

Verify the Data Guard configuration and ensure the standby database is receiving and applying redo:

DGMGRL> show database SALES_COPENHAGEN;
DGMGRL> show database SALES_AARHUS;
DGMGRL> validate database SALES_COPENHAGEN;
DGMGRL> validate database SALES_AARHUS;

That’s it.

Happy Patching!

Appendix

Static Listener Entry

If you use Oracle Restart or Oracle Grid Infrastructure, such entry is no longer needed.

How To Patch Oracle Data Guard Using AutoUpgrade And Standby-First Patch Apply With Restart

Let me show you how I patch my Oracle Data Guard configuration. I make it as easy as possible using Oracle AutoUpgrade. I reduce the interuption by doing standby-first patch apply with a primary database restart.

My Data Guard configuration consists of two databases:
- SID: SALES
- Databases: SALES_COPENHAGEN and SALES_AARHUS
- Hosts: copenhagen and aarhus
- Primary database: SALES_COPENHAGEN running on copenhagen

Preparations

You should do these preparations in advance of your maintenance window. They don’t interupt operations on your databases.

I download the patches using AutoUpgrade.
- I can download the patches from any computer. It doesn’t have to be one of the database hosts, which typically don’t have internet access.
I verify all patches are standby-first installable and my configuration meets the requirements for standby-first patch apply.

I create a new Oracle home on all hosts.

Create a config file called sales.cfg:

global.global_log_dir=/home/oracle/autoupgrade-patching/sales
patch1.source_home=/u01/app/oracle/product/19.3.0.0/dbhome_1
patch1.target_home=/u01/app/oracle/product/19/dbhome_19_26_0
patch1.sid=SALES
patch1.folder=/home/oracle/autoupgrade-patching/patch
patch1.patch=RECOMMENDED,MRP
patch1.download=no

Start AutoUpgrade in create_home mode:

java -jar autoupgrade.jar -config sales.cfg -patch -mode create_home

AutoUpgrade also runs root.sh if oracle user has sudo privileges. Else, I must manually execute root.sh.

Optionally, but recommended, I run an analysis on the primary database:
```
[oracle@copenhagen] java -jar autoupgrade.jar -config sales.cfg -patch -mode analyze
```
- Check the findings in the summary report.

Patching

Proceed with the following when your maintenance window starts.

Update listener.ora on the standby host (see appendix). I change the ORACLE_HOME parameter in the static listener entry (suffixed _DGMGRL) so it matches my target Oracle home.
I reload the listener:
```
[oracle@aarhus] lsnrctl reload
```

Patch the standby database:

[oracle@aarhus] java -jar autoupgrade.jar -config sales.cfg -mode deploy

I don’t disable redo transport/apply.

Optionally, test the application of patches using a snapshot standby database.
Downtime starts!
- Perform draining in advance according to your practices.
- Shut down your application.
Update listener.ora on the primary host (see appendix). I change the ORACLE_HOME parameter in the static listener entry (suffixed _DGMGRL) so it matches my target Oracle home.
I reload the listener:
```
[oracle@copenhagen] lsnrctl reload
```
Patch the primary database (see appendix):
```
[oracle@copenhagen] java -jar autoupgrade.jar -config sales.cfg -patch -mode deploy
```
- I use the sales.cfg config file.
- AutoUpgrade detects it’s running against the primary database, and executes Datapatch and all the post-upgrade tasks.

Verify the Data Guard configuration and ensure the standby database is receiving and applying redo:

DGMGRL> show database SALES_COPENHAGEN;
DGMGRL> show database SALES_AARHUS;
DGMGRL> validate database SALES_COPENHAGEN;
DGMGRL> validate database SALES_AARHUS;

That’s it.

Happy Patching!

Appendix

Static Listener Entry

If you use Oracle Restart or Oracle Grid Infrastructure, such entry is no longer needed.

AutoUpgrade New Features: Extended Platform Support

From the inception of Oracle AutoUpgrade, our clear ambition has been to support all platforms where you can run Oracle Database.

We still work with this ambition in mind, and the latest version of AutoUpgrade lifts additional platform restrictions. Now, you can enjoy AutoUpgrade’s true awesomeness on even more platforms.

Photo by Museums Victoria on Unsplash

Patching

You can now use one-button patching on the following platforms:

Linux x86-64
LINUX ARM
Microsoft Windows x64
Oracle Solaris on SPARC
Oracle Solaris on x86-64
IBM AIX on POWER Systems

How to Patch a Database

Ensure that you have the latest version of AutoUpgrade:

wget https://download.oracle.com/otn-pub/otn_software/autoupgrade.jar

Patch your database using the instructions from my previous blog post or our webinar.

How to Download Patches for Other Platforms

You can also use AutoUpgrade to just download patches from My Oracle Support and you can download for multiple platforms.

Ensure that you have the latest version of AutoUpgrade:

wget https://download.oracle.com/otn-pub/otn_software/autoupgrade.jar

Create a config file. Note there are two prefixes – one for Linux and one for Windows.

global.global_log_dir=/home/oracle/autoupgrade-patching/log
global.keystore=/home/oracle/autoupgrade-patching/keystore

patch1.folder=/home/oracle/autoupgrade-patching/patch
patch1.patch=RECOMMENDED
patch1.target_version=19
patch1.platform=LINUX.X64

patch2.folder=/home/oracle/autoupgrade-patching/patch
patch2.patch=RECOMMENDED
patch2.target_version=19
patch2.platform=WINDOWS.X64

Download the latest Release Update and other recommended patches for Linux and Windows:
```
java -jar autoupgrade.jar -config download.cfg -patch -mode download
```

You can tweak it even further:

Download for additional platforms by adding more prefixes. Check the documentation for supported platform names.
Download one-off patches or other patches by adding them to the patch parameter.

Some useful information:

If you omit platform it defaults to your current platform. If AutoUpgrade doesn’t support your current platform (like when you run AutoUpgrade on a Mac), it defaults to LINUX.X64.
Rodrigo Jorge from my team has a good blog post showing how to use download mode.

Windows

AutoUpgrade has supported upgrading on Windows right since the beginning – with the exception of Oracle RAC. But now, this restriction is lifted, so you can upgrade your Oracle RAC as well.

For patching, AutoUpgrade also supports Oracle RAC on Windows. But for now, it doesn’t perform rolling patching (nor does it do so on any platform). You still need opatchauto for that.

Instead, AutoUpgrade performs an all-node patching. It brings down all nodes at once, which means database downtime.

Linux 9

The latest release also includes a few bugs related to Oracle Linux 9. You should be able to use AutoUpgrade on this platform without problems.

Happy patching

How To Roll Back After Patching

Here’s a question I received from a customer:

I’ve patched my database to 19.25 using AutoUpgrade and out-of-place patching. How do I roll back, if needed?

Basically, a rollback is the same as patching the database. You just do it the other way around – from the higher to the lower Oracle home. But let’s look at the details.

AutoUpgrade

I’m glad to hear that the customer uses AutoUpgrade for patching. It’s my recommended method, and it has many benefits.

If you use AutoUpgrade to patch your Oracle Database, you can also use it to roll back, but only before going live:

java -jar autoupgrade.jar -restore -jobs <n>

n is the job ID of the patching job.
AutoUpgrade undoes everything it did.

AutoUpgrade relies on Flashback Database as its rollback mechanism. So, it’s no good if users have already connected to the database and added/changed data.

Allow me to repeat: Only use AutoUpgrade to roll back before go-live!

After go-live, you must roll back manually.

Manually

You can manually roll back at any time – even after go-live.

Imagine you want to roll back from 19.25 (the new Oracle home) to 19.24 (the old Oracle home). Here’s how to do it.

You start by setting the environment.

export OLD_ORACLE_HOME=/u01/app/oracle/product/dbhome_19_24
export NEW_ORACLE_HOME=/u01/app/oracle/product/dbhome_19_25
export ORACLE_HOME=$NEW_ORACLE_HOME
export PATH=$ORACLE_HOME/bin:$PATH

Optionally, you run Datapatch sanity check in the new Oracle home (thanks Erik for pointing that out).
```
$ORACLE_HOME/OPatch/datapatch -sanity_checks
```
You shut down the database running in the new Oracle home.
```
sqlplus / as sysdba<<EOF
   shutdown immediate
EOF
```

You move the following files back to the old Oracle home:

PFile
SPFile
Password file
Network files (like tnsnames.ora and sqlnet.ora)
Some of the files might not be present at all or be placed outside the Oracle home
Check this blog post for other files that might be stored in the Oracle home

mv $NEW_ORACLE_HOME/dbs/init$ORACLE_SID.ora $OLD_ORACLE_HOME/dbs
mv $NEW_ORACLE_HOME/dbs/spfile$ORACLE_SID.ora $OLD_ORACLE_HOME/dbs
mv $NEW_ORACLE_HOME/dbs/orapw$ORACLE_SID $OLD_ORACLE_HOME/dbs
mv $NEW_ORACLE_HOME/network/admin/sqlnet.ora $OLD_ORACLE_HOME/network/admin
mv $NEW_ORACLE_HOME/network/admin/tnsnames.ora $OLD_ORACLE_HOME/network/admin

Update /etc/oratab and set the Oracle home to the old one.
Update your profile scripts to reflect the old Oracle home. It could be .bashrc.

Start the database in the old Oracle home.

export ORACLE_HOME=$OLD_ORACLE_HOME
export PATH=$ORACLE_HOME/bin:$PATH
sqlplus / as sysdba<<EOF
   startup
EOF

Run Datapatch.
```
$ORACLE_HOME/OPatch/datapatch
```
Finally, you fix internal directories that point to paths inside the Oracle home:
```
@?/rdbms/admin/utlfixdirs.sql
```

Datapatch

When you roll back, you must execute Datapatch. It will automatically detect that you are rolling back and perform the necessary actions.

For each patch there is an apply script that brings changes into the database. Datapatch executes the apply script during patching.

For each apply script, there is always a rollback script. It will reverse the actions of the apply script. Datapatch executes the rollback script when you roll back.

You can learn much more about Datapatch in this video.

Normally, you would roll back to the Oracle home from where you came, but that’s not a requirement. This scenario is fully supported:

Patch from 19.23 to 19.25
Roll back to 19.24

How To Practice?

We have a hands-on lab in which you can try rollbacks – using AutoUpgrade and manually.

Patch Me If You Can

The lab runs in Oracle Live Labs.

It’s completely free
It runs in just a browser

Happy patching!

Grid Infrastructure 19c Out-Of-Place Patching Fails on AIX

I’m a strong advocate for out-of-place patching, and I can see that many of my blog readers are interested in that topic as well. Thank you for that!

But a reader notified me about a specific issue that occurs during out-of-place patching of Oracle Grid Infrastructure 19c. The issue occurs when using OPatchAuto as well as SwitchGridHome.

Normally, I recommend creating a new Oracle home using the base release (so 19.3.0) and then applying the latest Release Update on top:

# Unzipping base release, 19.3.0
unzip -oq /u01/software/LINUX.X64_193000_grid_home.zip
# Install and patch Oracle home
./gridSetup.sh -ignorePrereq -waitforcompletion -silent \
   -applyRU ...

However, that fails on AIX:

Preparing the home to patch...
Applying the patch /u01/software/36916690/36916690/36917416...
OPatch command failed while applying the patch. For details look at the logs 
from /u01/app/19.25.0/grid/cfgtoollogs/opatchauto/.

The log file has a little more detail:

DeleteAction : Destination File ''/u01/app/19.25.0/grid/perl/bin/perl'' is not writeable.
Copy Action: Destination File ''/u01/app/19.25.0/grid/perl/bin/perl'' is not writeable.

The Solution

There is already a MOS note that describes a potential workaround:

Out of place (OOP) patching of 19c Release Update (RU) fails on AIX (Doc ID 2948468.1)

But the reader leaving the comment asked for a few more words.

My Words

First, you should continue to use out-of-place patching despite the above issue.

Second, instead of using the base release (19.3.0) as the basis for any new Oracle home, you must create a new base release. One that doesn’t contain the error that leads to the above issue.

On a non-prod system, create a brand-new Grid Infrastructure installation using the base release (19.3.0).
Use in-place patching to patch it to the latest Release Update (currently 19.25.0). You need to add a few parameters to the opatchauto command:
```
<path_to_temp_home>/OPatch/opatchauto \
   apply <path-to-patch> \
   -binary \
   -oh <path_to_temp_home> \
   -target_type cluster
```

Create a gold image of this 19.25.0 Oracle home.

export NEW_GRID_HOME=/u01/app/19.25.0/grid
$NEW_GRID_HOME/gridSetup.sh -createGoldImage \
   -destinationLocation $GOLDIMAGEDIR \
   -name gi_gold_image.zip \
   -silent

You now have a new base release. It is almost as pristine as the 19.3.0 base release. It just contains the additional Release Update (19.3.0 + 19.25.0).
When you need to patch another system, use out-of-place patching using SwitchGridHome. But instead of using the base release 19.3.0, you use your new gold image that is already patched to 19.25.0.
```
#Don't do this
#unzip -oq /u01/software/LINUX.X64_193000_grid_home.zip
#Do this
unzip -oq /u01/software/gi_gold_image.zip
```
When you install the using gridSetup.sh you don’t have to apply the Release Update because the gold image contains it already. You can still apply any one-offs you need.
```
./gridSetup.sh -ignorePrereq -waitforcompletion -silent \
   -applyOneOffs <path_to_one_offs> \
   ...
```

There are no other changes to the procedure.

The issue is fixed in bug 34962446. However, I doesn’t seem to be available in 19c, so you have to repeat the above process for every Release Update.

If you still run into patching the Perl component, take a look at this MOS note:

[HAS]: Patching Got Failed with ‘DeleteAction : Destination File <GRID_HOME>/perl/bin/perl is not writeable’ ‘oracle.perlint, 5.28.1.0.0’ (Doc ID 2972916.1)

Final Words

Is it a viable workaround? Yes, I believe so. There’s a little more work to, on the other hard, you’ve now started to use gold images, which is a huge advantage.
If you continue patching in-place or out-of-place using OPatchAuto, be sure to clean up the Oracle home from time to time.
The issue occurs starting with Release Update 19.18 because that’s where Oracle started to add patches to Perl in the Oracle home.
Thanks to Axel Dellin for helping me with some details.

You should not let this little bump on the road prevent you from using out-of-place patching.

Happy Patching

The Basics

How to Patch the Listener

The Outage

Do I Need to Patch the Listener and Database At the Same Time?

What About AutoUpgrade?

How To Patch Oracle Database on Windows

The Methods

All At Once

Standby-first with restart

Standby-first with switchover

Summary

What If

RAC

Oracle Restart

We’re Really Sensitive To Downtime?

Datapatch

Appendix

Standby-First Installable

Other Blog Posts in the Series

Preparations

Patching

Post-Patching

Appendix

Static Listener Entry

Further Reading

Other Blog Posts in the Series

How To

Is It Safe?

What Happens If I Need to Switch Over or Fail Over?

What Happens If Datapatch Fails

Appendix

Other Blog Posts in the Series

Preparations

Patching

Appendix

Static Listener Entry

Further Reading

Other Blog Posts in the Series

Preparations

Patching

Appendix

Static Listener Entry

Further Reading

Other Blog Posts in the Series

Patching

How to Patch a Database

How to Download Patches for Other Platforms

Windows

Linux 9

AutoUpgrade

Manually

Datapatch

How To Practice?

Further Reading

The Solution

My Words

Final Words