Category: Patching

Patching Eats Your SYSTEM Tablespace

Everyone says: Patch your software!

I keep saying: Patch your Oracle Database!

Let me tell you a little secret. All that patching is eating space in your SYSTEM tablespace…

This blog post is a continuation of Mike’s many blog posts on the topic. It adds an example with some real numbers.

Apply And Rollback Scripts

Normally, the apply and rollback scripts are stored in the Oracle home. Here’s an example:

$ ll $ORACLE_HOME/sqlpatch
drwxr-xr-x. 4 oracle oinstall     38 Apr 18  2019 29517242
drwxr-xr-x. 3 oracle oinstall     22 Jul 28 05:28 36878697
drwxr-xr-x. 3 oracle oinstall     22 Jul 28 05:27 36912597
drwxr-xr-x. 3 oracle oinstall     22 Jul 28 05:30 37056207
drwxr-xr-x. 3 oracle oinstall     22 Aug 11 08:34 37102264
drwxr-xr-x. 3 oracle oinstall     22 Aug 11 08:32 37260974
drwxr-xr-x. 3 oracle oinstall     22 Aug 11 08:36 37470729
drwxr-xr-x. 3 oracle oinstall     22 Aug 11 08:47 37499406
drwxr-xr-x. 3 oracle oinstall     22 Aug 11 08:44 37642901
drwxr-xr-x. 3 oracle oinstall     22 Aug 11 08:49 37777295

When Datapatch applies patches to your Oracle Database, it uses the apply and rollback scripts from the Oracle home.

But during apply, Datapatch also takes the rollback scripts and stores them inside the database – in the SYSTEM tablespace. This ensures that Datapatch can always roll back patches, regardless of whether the rollback scripts are in the Oracle home.

When you use the OPatch cleanup or uses out-of-place patching, there is a risk that Datapatch might need a rollback script which is no longer found in the Oracle home. But then Datapatch simply finds it in the SYSTEM tablespace instead.

This avoild a lot of chaotic situations with missing rollback scripts.

Storing The Scripts

Datapatch uses two tables that both have a column named PATCH_DIRECTORY:

select * from dba_registry_sqlpatch;
select * dba_registry_sqlpatch_ru_info;

This query shows each patch action and corresponding usage for apply/rollback script:

select * from (
   select description, round(dbms_lob.getlength(PATCH_DIRECTORY)/1024/1024, 2) as size_mb
   from dba_registry_sqlpatch
   where action='APPLY' and description not like 'Database Release Update%'
   union
   select 'Release Update ' || RU_version as description, round(dbms_lob.getlength(PATCH_DIRECTORY)/1024/1024) as size_mb
   from dba_registry_sqlpatch_ru_info)
order by description;

Here’s an example of an Oracle Database that I have patched a few times.

DESCRIPTION                                       SIZE_MB
------------------------------------------------- -------
DATAPUMP BUNDLE PATCH 19.25.0.0.0                    1.03
DATAPUMP BUNDLE PATCH 19.26.0.0.0                    1.03
DATAPUMP BUNDLE PATCH 19.27.0.0.0                    1.03
DATAPUMP BUNDLE PATCH 19.28.0.0.0                    1.04
OJVM RELEASE UPDATE: 19.25.0.0.241015 (36878697)      .01
OJVM RELEASE UPDATE: 19.26.0.0.250121 (37102264)      .02
OJVM RELEASE UPDATE: 19.27.0.0.250415 (37499406)      .02
OJVM RELEASE UPDATE: 19.28.0.0.250715 (37847857)      .02
Release Update 19.25.0.0.0                            175
Release Update 19.26.0.0.0                            184
Release Update 19.27.0.0.0                            194
Release Update 19.28.0.0.0                            203
Release Update 19.3.0.0.0                               4

10 rows selected.

That’s around 750 MB.

Cleaning Up

Datapatch only needs the rollback scripts for the patches that are currently applied. You can remove all other scripts:

$ORACLE_HOME/OPatch/datapatch -purge_old_metadata

Using the above environment, this is the result of the cleanup:

DESCRIPTION                                       SIZE_MB
------------------------------------------------- -------
DATAPUMP BUNDLE PATCH 19.25.0.0.0                    
DATAPUMP BUNDLE PATCH 19.26.0.0.0                    
DATAPUMP BUNDLE PATCH 19.27.0.0.0                    
DATAPUMP BUNDLE PATCH 19.28.0.0.0                    1.04
OJVM RELEASE UPDATE: 19.25.0.0.241015 (36878697)     
OJVM RELEASE UPDATE: 19.26.0.0.250121 (37102264)     
OJVM RELEASE UPDATE: 19.27.0.0.250415 (37499406)     
OJVM RELEASE UPDATE: 19.28.0.0.250715 (37847857)      .02
Release Update 19.25.0.0.0                           
Release Update 19.26.0.0.0                           
Release Update 19.27.0.0.0                           
Release Update 19.28.0.0.0                            203
Release Update 19.3.0.0.0                               

10 rows selected.

Datapatch now only uses little more than 200 MB.

A few comments about the cleanup functionality:

  • It is available via patch 37738908. Hopefully, it will be part of the 19.29 Release Update.
  • You can safely execute the cleanup. Datapatch doesn’t remove scripts that it might need at a later point.
  • Oracle recommends that you run the cleanup in an off-peak period.
  • The cleanup happens via a TRUNCATE TABLE command which effectively reclaims space so other segments may use it. However, it doesn’t shrink the tablespace, so the physical size of the data files remain the same.
  • To facilitate the TRUNCATE TABLE command, those records that must remain is copied to a new table. After truncating the original table, those records are moved back and the temporary table is dropped. This might lead to a little increase in space usage while Datapatch cleans up.
  • When you upgrade, the upgrade engine truncates those tables. They are of no use following an upgrade.

Final Words

Check your Oracle Database. How much space does Datapatch use? What was the largest amount of space you could reclaim? Let me know in the comments below.

Update

  • 03-SEP-2025: I correctly wrote that the tables use the SYSAUX tablespace. They use the SYSTEM tablespace. Also, I added a detail about the cleanup might take up a little more space temporarily. Thanks to Pete for letting me know.

AutoUpgrade New Features: Create Oracle Home Path Dynamically

Some customers prefer to install Oracle homes in a path that indicates the patch level, like:

/u01/app/oracle/dbhome_1928

1928 indicates this is Oracle Database 19c with Release Update 28. If you use AutoUpgrade to create your Oracle home, you can specify that path in your config file:

patch1.target_home=/u01/app/oracle/dbhome_1928

To ease automation, AutoUpgrade can dynamically determine the target Oracle home using placeholders. After downloading the patches, it checks which Release Update it will apply.

patch1.target_home=/u01/app/oracle/dbhome_%RELEASE%%UPDATE%

Easier Patching

Take a look at this config file

global.global_log_dir=/home/oracle/autoupgrade-patching/log
global.keystore=/home/oracle/autoupgrade-patching/keystore
global.folder=/home/oracle/patch-repo
patch1.source_home=/u01/app/oracle/product/dbhome_1927
patch1.target_home=/u01/app/oracle/product/dbhome_%RELEASE%%UPDATE%
patch1.sid=FTEX
patch1.patch=RECOMMENDED
  • patch=RECOMMENDED or even patch=RU would always get the latest Release Update. At time of writing, that would be 19.28.
  • target_home contains the placeholders. After AutoUpgrade determined that 19.28 is the latest Release Update, it would dynamically determine the Oracle home path.
  • You would still need to update source_home in the config file. It will change after each run because you use out-of-place patching.

Now, you can just start AutoUpgrade every quarter:

java -jar autoupgrade.jar \
     -config FTEX.cfg \
     -patch \
     -mode deploy
  • AutoUpgrade downloads the latest Release Update and builds a new Oracle home for your – with a dynamically created path reflecting the Release Update.

Ain’t that easy?

Other Release Updates

What if you want to install an older Release Update. That works too:

patch1.target_home=/u01/app/oracle/product/dbhome_%RELEASE%_%UPDATE%
patch1.patch=RU:19.27,OPATCH

This will install the Oracle home in:

/u01/app/oracle/product/dbhome_19_27

Even Easier

Here’s another way. It requires that the environment is set correctly and works entirely without a config file.

export ORACLE_SID=FTEX
export ORACLE_HOME=/u01/app/oracle/product/19
export AU_LOG_DIR="/home/oracle/autoupgrade-patching/log_"`date +%Y%m%d_%H%M%S`
java -jar autoupgrade.jar \
     -config_values "global.global_log_dir=$AU_LOG_DIR,global.folder=/home/oracle/patch-repo,download=no,target_home=/u01/app/oracle/product/dbhome_%RELEASE%%UPDATE%,patch=RECOMMENDED" \
     -patch \
	 -mode deploy
  • AutoUpgrade takes sid and source_home from the environment variables ORACLE_SID and ORACLE_HOME.
  • Then you specify other parameters as a comma-separated list using -config_values.
  • Notice the target_home with placeholders in the long list of parameters.
  • The AutoUpgrade logging directory uses a timestamp to ensure a new one is used on each invocation.

Final Words

It’s now easier to automate creation of Oracle homes using AutoUpgrade.

Do you think this feature is useful? Do you need other replacement variables?

Leave a comment below and let me know.

AutoUpgrade New Features: Patch OCW Component In Oracle Home

Every Oracle home contains an Oracle Clusterware (OCW) component. It’s used to interact with Grid Infrastructure when you are using Oracle Restart or Oracle RAC. But even when you don’t use those, the component is still part of your Oracle home.

The Database Release Update doesn’t update the OCW component in your Oracle home. You must use the Grid Infrastructure Release Update for that.

In AutoUpgrade, it is easy to update the OCW component. Let’s see how it works.

How To Also Patch The OCW Component

  • My database hasn’t been patched for a while:

    $ORACLE_HOME/OPatch/opatch lspatches

35648110;OJVM RELEASE UPDATE: 19.21.0.0.231017 (35648110)
35787077;DATAPUMP BUNDLE PATCH 19.21.0.0.0
35643107;Database Release Update : 19.21.0.0.231017 (35643107)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
    • I’ve never updated the OCW component, so it’s still on the patch level of the base release, 19.3.0.0.0.

  • I use the latest version of AutoUpgrade:

    wget https://download.oracle.com/otn-pub/otn_software/autoupgrade.jar

  • I create an AutoUpgrade config file, FTEX.cfg:

    global.global_log_dir=/home/oracle/autoupgrade-patching/log
global.keystore=/home/oracle/autoupgrade-patching/keystore
patch1.source_home=/u01/app/oracle/product/19
patch1.target_home=/u01/app/oracle/product/19_27
patch1.sid=FTEX
patch1.folder=/home/oracle/patch-repo
patch1.patch=OPATCH,RU,OCW,DPBP,OJVM
    • By adding OCW to the patch parameter, AutoUpgrade also downloads the GI Release Update and updates the OCW component.

  • I patch the database:

    java -jar autoupgrade.jar -config FTEX.cfg -patch -mode deploy

  • When AutoUpgrade completes, I check the new patch level:

    $ORACLE_HOME/OPatch/opatch lspatches

37499406;OJVM RELEASE UPDATE: 19.27.0.0.250415 (37499406)
37654975;OCW RELEASE UPDATE 19.27.0.0.0 (37654975)
37777295;DATAPUMP BUNDLE PATCH 19.27.0.0.0
37642901;Database Release Update : 19.27.0.0.250415 (37642901)
    • Notice how the OCW Release Update is now 19.27.0.0.0.

Some Details

  • When AutoUpgrade downloads patches, because I specified OCW, it will also download the GI Release Update:

     --------------------------------------------
 Downloading files to /home/oracle/patch-repo
 --------------------------------------------
 DATABASE RELEASE UPDATE 19.27.0.0.0
     File: p37642901_190000_Linux-x86-64.zip - LOCATED
 
 DATAPUMP BUNDLE PATCH 19.27.0.0.0
     File: p37777295_1927000DBRU_Generic.zip - LOCATED
 
 GI RELEASE UPDATE 19.27.0.0.0
     File: p37641958_190000_Linux-x86-64.zip / 83%

  • Including OCW is a smart way of downloading the GI Release Update. You can use it to patch your Grid Infrastructure.

  • In Oracle Database 23ai, you can download fully updated gold images. Besides having the latest Release Update, they also come with fully updated OCW components.

Is It Needed?

Should you update the OCW component when you patch your Oracle Database? Is it needed if you don’t use Oracle Restart, Oracle RAC, or Oracle ASM?

It is optional, but even if no GI Stack (ASM, Clusterware or RAC) is used inside the server, it is recommended not to ignore the security patches of the installed components. And apply the most recent OCW Patch.

How to apply OCW Release Update patches on db_home non-RAC / non-ASM (Doc ID 2970542.1)

Mike Dietrich has a good point as well:

As I neither use RHP/FPP or any of the HA components nor EM in my tiny little lab environments, I’m pretty certain that I won’t need the OCW bundle. But this may be different in your environments. And it doesn’t harm to apply it of course.

Adding the Oracle 19.14.0 OCW / GI bundle patch to my database home

Further, I know many customers who never patch the OCW component and haven’t run into related problems.

My recommendation: Update the OCW component when you patch your Oracle Database. Using AutoUpgrade it is so easy, that there’s no reason not to.

Happy patching!

AutoUpgrade New Features: Custom Oracle Home Name

When you create a new Oracle home using AutoUpgrade, you can now give it a custom name.

What do I mean by a custom name? Let’s examine the Oracle Inventory:

cat /u01/app/oraInventory/ContentsXML/inventory.xml

<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2025, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>12.2.0.7.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraDB19Home1" LOC="/u01/app/oracle/product/19" TYPE="O" IDX="1"/>
<HOME NAME="OraDB21Home1" LOC="/u01/app/oracle/product/21" TYPE="O" IDX="2"/>
<HOME NAME="OraDB23Home1" LOC="/u01/app/oracle/product/23" TYPE="O" IDX="3"/>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>

The Oracle home name is listed in the NAME attribute on the HOME element.

When you install a new Oracle home, the installer automatically generate a name for you, like OraDb19Home1.

In AutoUpgrade, you can decide to use a custom name instead of an auto-generated one.

How To Specify A Custom Oracle Home Name

  • I use the latest version of AutoUpgrade
    wget https://download.oracle.com/otn-pub/otn_software/autoupgrade.jar
  • I create the following config file:
    global.global_log_dir=/home/oracle/autoupgrade-patching/log
global.keystore=/home/oracle/autoupgrade-patching/keystore
patch1.source_home=/u01/app/oracle/product/19
patch1.target_home=/u01/app/oracle/product/19_28
patch1.home_settings.home_name=DBHOME1928
patch1.sid=FTEX
patch1.folder=/home/oracle/patch-repo
patch1.patch=RU,OCW,DPBP,OJVM,OPATCH
    • Notice the home_settings.home_name parameter. This is where I specify the custom name for the new Oracle home.
  • I patch the database using AutoUpgrade:
    java -jar autoupgrade.jar -config FTEX.cfg -mode deploy
  • I check the inventory:
    grep -i "19_28" /u01/app/oraInventory/ContentsXML/inventory.xml
<HOME NAME="DBHOME1928" LOC="/u01/app/oracle/product/19_28" TYPE="O" IDX="4"/>
    • AutoUpgrade created the new Oracle home with the name DBHOME1928.

What Do I Use It For

For a regular Oracle home, it’s used seldom. I’ve seen some customers using certain Oracle home names as part of their corporate standard, but most people don’t care.

However, in a read-only Oracle home, the name is important. It now becomes part of Oracle Base Home which is the location for most of the writable files from the Oracle home.

Do you have a specific use case for the Oracle home name or do you have a corporate standard mandating a certain name? Leave a comment and let me know.

Happy patching!

The Easiest Way To Download Patches for Oracle Grid Infrastructure

Whether you patch Oracle Grid Infrastructure manually, using your own automation, or Oracle Fleet Patching and Provisioning, it all starts with downloading the patches.

Although AutoUpgrade doesn’t patch Grid Infrastructure, it can still download the latest version of OPatch and the GI Release Update.

How To Download Grid Infrastructure Release Update

I have already configured the AutoUpgrade keystore and saved my My Oracle Support credentials. You find instructions on how to do that here (search for Creating an AutoUpgrade Keystore).

  1. I download the latest version of AutoUpgrade:

    wget https://download.oracle.com/otn-pub/otn_software/autoupgrade.jar

  2. I create a config file called get-gi-patches.cfg:

    global.global_log_dir=/home/oracle/autoupgrade/logs
global.keystore=/home/oracle/autoupgrade/keystore

patch1.patch=RU,OPATCH,OCW
patch1.target_version=19
patch1.platform=LINUX.X64
patch1.folder=/home/oracle/autoupgrade/patches

  3. I create the folder for the patches (config file parameter folder).

    mkdir -p /home/oracle/autoupgrade/patches

  4. I download the patches by starting AutoUpgrade in download mode:

    java -jar autoupgrade.jar -config get-gi-patches.cfg -mode download

  5. Here’s the output from AutoUpgrade:

    AutoUpgrade Patching 25.3.250509 launched with default internal options
Processing config file ...
Loading AutoUpgrade Patching keystore
AutoUpgrade Patching keystore is loaded

Connected to MOS - Searching for specified patches

-----------------------------------------------------
Downloading files to /home/oracle/autoupgrade/patches
-----------------------------------------------------
DATABASE RELEASE UPDATE 19.27.0.0.0
    File: p37642901_190000_Linux-x86-64.zip - VALIDATED

GI RELEASE UPDATE 19.27.0.0.0
    File: p37641958_190000_Linux-x86-64.zip - VALIDATED

OPatch 12.2.0.1.46 for DB 19.0.0.0.0 (Apr 2025)
    File: p6880880_190000_Linux-x86-64.zip - VALIDATED
-----------------------------------------------------

  6. That’s it! After a few minutes, I’ve downloaded the latest GI Release Update and OPatch. GI RELEASE UPDATE 19.27.0.0.0 – p37641958_190000_Linux-x86-64.zip OPatch 12.2.0.1.46 for DB 19.0.0.0.0 (Apr 2025) – p6880880_190000_Linux-x86-64.zip

Is it really that easy? Yes, it is…

I can now patch my GI Oracle home – for Oracle RAC Database and Oracle Restart.

Happy patching!

Appendix

I Want To Download A Previous Release Update

Rather than downloading the latest Release Update, you can choose to download a specific Release Update. You can specify that in the patch parameter:

patch1.patch=RU:19.26,OPATCH,OCW
  • Notice how the RU keyword has a suffix specifying the exact Release Update.

Fixing AutoUpgrade PATCH111 Error On Oracle Linux 9

I’ve seen a lot of interest in the create_home mode of AutoUpgrade. It allows you to easily provision new Oracle homes and the feedback has been very positive.

A few customers reported an error during installation of an Oracle home on Oracle Linux 9.

Errors in database [create_home_1]
Stage              [OH_PATCHING]
Operation          [STOPPED]
Status             [ERROR]
Info               [Exception: PATCH111
Err message: AutoUpgradePatchingException [OPatch failed to install ...]

Digging into the referenced log file revealed:

[Jun 25, 2025 10:48:51 AM] [WARNING]OUI-67200:Make failed to invoke "/usr/bin/make -f ins_rdbms.mk javavm_refresh ORACLE_HOME=/u01/app/oracle/product/dbhome_1927 OPATCH_SESSION=apply"....'Can't locate File/Copy.pm in @INC (you may need to install the File::Copy module) (@INC contains: /usr/local/lib64/perl5/5.32 /usr/local/share/perl5/5.32 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at /u01/app/oracle/product/dbhome_1927/javavm/install/update_javavm_binaries.pl line 64.
                                    BEGIN failed--compilation aborted at /u01/app/oracle/product/dbhome_1927/javavm/install/update_javavm_binaries.pl line 64.
                                    make: *** [ins_rdbms.mk:573: javavm_refresh] Error 2

The Workaround

You can avoid the problem by installing Perl before you start AutoUpgrade:

dnf -y install perl-core

Big shoutout to Martin Berger for providing a workaround.

Is My Server Affected

Use this command to test whether your server is affected:

perl -e 'use File::Copy;'

If it returns without any error or output, everything is fine. If you get an error, you must install Perl before proceeding:

perl -e 'use File::Copy;'
Can't locate File/Copy.pm in @INC (you may need to install the File::Copy module) (@INC contains: /usr/local/lib64/perl5/5.32 /usr/local/share/perl5/5.32 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at -e line 1.
BEGIN failed--compilation aborted at -e line 1.

Why Does It Happen

  • AutoUpgrade starts by installing the Oracle home and in some cases also applies the Release Update using the -applyRU command line parameter.

  • Next, it installs additional patches using OPatch. Some patches, like the OJVM bundle patch, requires a compilation using make. This process fails if there’s no usable Perl installation.

  • There are several MOS notes on this specific issues advising to set the PERL5LIB environment variable (Patching fails during relink , with error code 102 :: Fatal error: Command failed for target `javavm_refresh’ (Doc ID 2002334.1)).

  • To solve an unrelated issue, AutoUpgrade doesn’t always set PERL5LIB when calling OPatch and this leads to this error during the compilation.

  • We’re trying to find a smart way to handle this (without re-introducing the other issue).

To summarize, you get this error when:

  • Your operating system doesn’t have a normal Perl installation. You might have the perl executable, but not the normal modules.
  • You install the OJVM bundle patch.

The Rabbit Hole

I learned something new about Perl today. Do you dare walking down that rabbit hole with me? I’ll make it short.

  • The error from make mentions the missing Perl module, File::Copy:

    /usr/bin/make -f ins_rdbms.mk javavm_refresh ORACLE_HOME=/u01/app/oracle/product/dbhome_1927 OPATCH_SESSION=apply"....'Can't locate File/Copy.pm in @INC (you may need to install the File::Copy module)

  • A Perl module can be loaded from disk. Perl automatically attempts to find the file in the following locations:

    perl -e 'print "@INC\n"'
/usr/local/lib64/perl5/5.32
/usr/local/share/perl5/5.32
/usr/lib64/perl5/vendor_perl
/usr/share/perl5/vendor_perl
/usr/lib64/perl5
/usr/share/perl5

  • So, I can check these directories for the Perl module. To find the File::Copy module, I need to check in the File subdirectory for the file Copy.pm:

    ll /usr/local/lib64/perl5/5.32/File/Copy.pm
ll /usr/local/share/perl5/5.32/File/Copy.pm
ll /usr/lib64/perl5/vendor_perl/File/Copy.pm
ll /usr/share/perl5/vendor_perl/File/Copy.pm
ll /usr/lib64/perl5/File/Copy.pm
ll /usr/share/perl5/File/Copy.pm
ls: cannot access '/usr/local/lib64/perl5/5.32/File/Copy.pm': No such file or directory
ls: cannot access '/usr/local/share/perl5/5.32/File/Copy.pm': No such file or directory
ls: cannot access '/usr/lib64/perl5/vendor_perl/File/Copy.pm': No such file or directory
ls: cannot access '/usr/share/perl5/vendor_perl/File/Copy.pm': No such file or directory
ls: cannot access '/usr/lib64/perl5/File/Copy.pm': No such file or directory
ls: cannot access '/usr/share/perl5/File/Copy.pm': No such file or directory

  • The file is not found in any of the directories, so Perl will fail to load the module.

  • Had the file been found in just one location, all would be good.

Happy patching!

Where Is The OJVM Bundle Patch For Oracle Database 23ai?

A customer asked me:

Where do I download the OJVM bundle patch for Oracle Database 23ai? I can’t find it anywhere.

In Oracle Database 23ai, Oracle ships fully updated gold images instead of individual Release Updates. The customer installed the latest gold image and it didn’t contain the OJVM bundle patch:

$ORACLE_HOME/OPatch/opatch lspatches
37701424;OCW RELEASE UPDATE 23.8.0.25.04 (37701424) Gold Image
37701421;Database Release Update : 23.8.0.25.04 (37701421) Gold Image

In Oracle Database 19c, you would be able to see the OJVM bundle patch as a separate patch:

$ $ORACLE_HOME/OPatch/opatch lspatches
35648110;OJVM RELEASE UPDATE: 19.21.0.0.231017 (35648110)
35643107;Database Release Update : 19.21.0.0.231017 (35643107)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

Why is it missing from Oracle Database 23ai gold image and why can’t you download it separately?

OJVM In Oracle Database 23ai

Starting in Oracle Database 21c, there is no longer a separate OJVM bundle patch. Instead, the fixes are fully included in the Release Updates.

… the OJVM component patch has been incorporated within the 21c Database Release Update (RU) patch

Oracle Recommended Patches – “Oracle JavaVM Component Database PSU and Update” (OJVM PSU and OJVM Update) Patches (Doc ID 1929745.1)

Also, there is no longer any interuption or brownout when you start Datapatch. The OJVM fixes are now fully RAC Rolling Installable and Standby-First Installable.

So, just get the newest gold image in Oracle Database 23ai and that’s it.

What About Oracle Database 19c

Let’s just recap the situation when you have OJVM in Oracle Database 19c.

  • The patch is RAC Rolling Installable
  • Standby-First Installable (as long as you don’t have Active Data Guard and extensively use OJVM on the standby database)

First, are you using OJVM or can you remove it?

Then, how do you patch OJVM?

  1. Start by downloading the OJVM bundle patch.
    1. Use AutoUpgrade and include the OJVM keyword in your patch specification, like patch1.patch=OPATCH,RU,OJVM
    2. Use the Download Assistant on My Oracle Support
  2. When you start Datapatch and it applies the OJVM fixes, there is a short brownout while the Java system is reloaded. Any session that tries to use Java at that point will get ORA-29548 but will be able to retry after a few seconds.

Here’s a little video with additional information.

What About Grid Infrastructure?

You should not apply the OJVM bundle patch to your Grid Infrastructure homes.

OJVM PSU is not needed in the Grid home, only in the database home.

Oracle Recommended Patches – “Oracle JavaVM Component Database PSU and Update” (OJVM PSU and OJVM Update) Patches (Doc ID 1929745.1)

Happy patching!

AutoUpgrade New Features: Run root.sh After Oracle Home Creation

AutoUpgrade relies on out-of-place patching, thus, it creates a new Oracle home. After installing a new Oracle home, you must execute root.sh as the root user.

AutoUpgrade now does this for you under the following circumstances:

  • oracle user has sudo privileges.
  • The sudo command doesn’t require a password.

AutoUpgrade handles root.sh in the ROOTSH stage, and writes log messages to <autoupgrade_log_dir>/<sid>/<job-no>/rootsh/rootsh.log.

Sudo Privileges

If oracle has sudo privileges, no configuration is needed. AutoUpgrade sees this and automatically executes root.sh after installing the Oracle home and applying the desired patches.

What If

If AutoUpgrade can’t execute root.sh, you must do it manually. AutoUpgrade writes a message into the rootsh.log:

2025-03-24 07:39:22.297 INFO Root Script Path : /u01/app/oracle/product/19_25/root.sh - RootScript.executePatchStage#66
2025-03-24 07:39:22.305 INFO Executing Root Script - RootScript.updateAndLogInfoMessage#91
sudo: a password is required
2025-03-24 07:39:22.327 INFO Root script execution failed sudo: a password is required
 - RootScript.executeRootScript#112
2025-03-24 07:39:22.327 INFO AutoUpgrade Patching has not run /u01/app/oracle/product/19_25/root.sh for the newly installed ORACLE_HOME. This needs to be performed manually after AutoUpgrade Patching completes. - RootScript.executePatchStage#83

If AutoUpgrade can’t run the root.sh for you, you shouldn’t use the one-button approach, where it handles everything in one operation. Instead, you should follow this approach:

  • Install the new Oracle home using -mode create_home.

  • Manually run root.sh.

  • Patch the database using -mode deploy. AutoUpgrade sees that you’ve already created the new Oracle home, and skips that part.

Why do you need to do it in multiple steps?

Some functionality in the database requires that you run root.sh. Also, there are certain patches that require you to run root.sh before starting the database. One example is patch 34672698, ORA-00800: SOFT EXTERNAL ERROR, ARGUMENTS: [SET PRIORITY FAILED], [VKTM] , DISM(16). Suppose you haven’t set the proper file permissions with root.sh before starting the database, the database won’t be able to elevate the priority of some of the most critical background processes.

The Special Patches

As mentioned above, there are some special patches that require that you fiddle with file system permissions during the apply.

One example is patch 34672698, ORA-00800: SOFT EXTERNAL ERROR, ARGUMENTS: [SET PRIORITY FAILED], [VKTM] , DISM(16). Check the readme to see the instructions explaining how to change permissions on $ORACLE_HOME/bin/oradism.

Using AutoUpgrade, installing such a patch is very easy. AutoUpgrade takes all that into consideration. This is how AutoUpgrade creates the new Oracle home:

  • Unzip the Oracle home.
  • Installs the Oracle home using runInstaller, but doesn’t run root.sh.
  • Applies patches using OPatch, including such special patches.
  • Executes root.sh, which applies all the required file permissions.

The execution of root.sh in the end takes care of all the extra work with permissions.

Happy patching!

How to Patch Oracle Net Listener

When running Oracle Database on a server, the Oracle Net Listener is a vital component.

Let’s discuss how you can patch it.

The Basics

Normally, on a server, you have:

  • One listener
  • One or many database instances

That one listener handles new connections to all database instances. You patch the listener simply by starting it in the new Oracle home.

You can have multiple listeners and use remote listeners, but if you do that, you’re smart and probably won’t need the advice in this blog post.

How to Patch the Listener

  1. Copy listener.ora from the old Oracle home to the new Oracle home.
    export OLD_ORACLE_HOME=/u01/app/oracle/product/19.26.0/dbhome_1
export NEW_ORACLE_HOME=/u01/app/oracle/product/19.27.0/dbhome_1
cp $OLD_ORACLE_HOME/network/admin/listener.ora $NEW_ORACLE_HOME/network/admin
    • You can avoid this step by redirecting the location of your network files using the TNS_ADMIN environment variable.
    • If you have a completely default installation with no listener configuration, you don’t have a listener.ora file and can skip this step.
    • In an advanced configuration you might have references to the listener Oracle home. Those you must update. But don’t confuse the listener Oracle home, with the Oracle homes of the databases with static registration. Those you update, when you patch the matching database.
  2. Restart the listener in the new Oracle home:
    export ORACLE_HOME=$OLD_ORACLE_HOME
$OLD_ORACLE_HOME/bin/lsnrctl stop
export ORACLE_HOME=$NEW_ORACLE_HOME
$NEW_ORACLE_HOME/bin/lsnrctl start
    • If you use Oracle Grid Infrastructure or Oracle Restart, they manage the listener. You automatically restart, and thus patch, the listener when you patch those components.
  3. If you use the dbstart script to start the listener, update it with the new Oracle home.
    • On Windows, you must delete the matching Windows service when you stop the listener. The lsnrctl start command should create a new listener. Be sure to set it to start automatically.

The Outage

  • When you restart the listener, there’s a short period where there’s no listener. It takes a second or so for the new listener to start. In that period, any new connection attempt fails:

    ORA-12541: TNS:no listener

    This affects new connections only. Existing connections continue to work unaffected.

  • As soon as the listener starts, it’ll read listener.ora and go through the static listener registrations defined by the SID_LIST. Connections to those databases are ready immediately.

  • For optimal flexibility, however, it’s better to use dynamic listener registration, where the database instance automatically registers with the listener. When you restart the listener, it doesn’t know of all the dynamic registrations and will have to wait until the database registers (up to 60 seconds).

    • You can force a database to register with the listener:
      SQL> alter system register;
    • It’s a good idea to run through all of your databases after the listener restart and force a listener registration with the above command.
    • Until the dynamic registration completes, any new connection to the database will fail:
      ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

  • You can hide the short outage from the client by using RETRY_COUNT and RETRY_DELAY in your connect strings. Allow the client to retry for a short period before throwing the above errors.

  • You can avoid the outage completely by using multiple listeners and patching them separately. Your connect strings should have an ADDRESS_LIST containing both listeners.

Do I Need to Patch the Listener and Database At the Same Time?

No, that’s up to you. If you want just one interruption, then do it all at once. Otherwise, you can do it in separate maintenance windows.

What About AutoUpgrade?

At the moment, AutoUpgrade doesn’t patch the listener. But it’s on the backlog.

Happy patching!

How to Patch Oracle Database on Exadata Cloud@Customer using Out-Of-Place

At our recent workshop in London, I was advocating for using out-of-place patching and the many benefits it brings. One of the questions was:

How do I do that on Exadata Cloud@Customer?

When you use the patching functionality in the cloud tooling, it will patch an entire Oracle home, including all databases, using in-place patching. This is a convenient option, but it extends the maintenance window on your database.

Let’s find out how to do it using out-of-place patching!

Benefits of Out-Of-Place Patching

  • You can patch the database with less downtime. You just shut down the database in the old Oracle home and immediately restart it in the new Oracle home.
    • With in-place patching, you need more downtime. While the database is down, you need time to roll off patches and apply the new ones. Depending on the number of patches you have, this can take 5, 10, or 20 minutes.
  • You prepare the new Oracle home in advance.
    • With in-place patching, all changes to the Oracle home occur during the maintenance window when the database is down.
  • There are fewer tasks in your downtime window, so less risk and less stress.
  • You can use a fresh, brand-new Oracle home to avoid issues with rolling off patches. Plus, a brand-new Oracle home means you don’t carry any old luggage in your Oracle home.
    • With in-place patching, your Oracle home will grow over time. You’ll see that disk space keeps increasing.
  • In the unlikely event that you need to roll back or fall back, that’s a lot easier. You just restart the database in the old Oracle home.
    • With in-place patching, you have to reverse the entire process. So, all that was cumbersome when you patched will be cumbersome when you roll back.

AutoUpgrade

In the workshop, I showed some cool demos of using AutoUpgrade for out-of-place patching. You can’t do that on Exadata Cloud@Customer. You must use the cloud tooling when you patch an entire CDB.

Alternatively, you can patch an individual PDB using AutoUpgrade and refreshable clone PDBs. I’ll have to write a blog post about that.

How to Patch Out-Of-Place on Exadata Cloud@Customer

  1. Create a software image at the desired Release Update with any additional one-off patches. You can skip this if you want to use the Oracle-provided images.
  2. Create a database home using the software image you just created, or use an Oracle-provided one.
  3. I recommend that you update the cloud tooling.
  4. Your maintenance window starts.
  5. Move the database to the new Oracle home. The cloud tooling does that in a rolling manner, so there’s no database outage.
    • It also executes Datapatch and other post-patching steps.

That’s it; your database has been patched!

  • After a while, you can remove the old Oracle home.
  • If you want to script the process, you can also use the REST APIs or dbaascli.
  • This also works for Exadata Database Service.
  • There’s no way to use out-of-place patching on Grid Infrastructure.

Happy patching!

Appendix

Other Methods

You are free to use Exadata Fleet Update for your Exadata Cloud@Customer system. It can greatly reduce your workload and automate your patching operations. Check out these resources:

Further Reading