Tag: ojvm

Fixing AutoUpgrade PATCH111 Error On Oracle Linux 9

I’ve seen a lot of interest in the create_home mode of AutoUpgrade. It allows you to easily provision new Oracle homes and the feedback has been very positive.

A few customers reported an error during installation of an Oracle home on Oracle Linux 9.

Errors in database [create_home_1]
Stage              [OH_PATCHING]
Operation          [STOPPED]
Status             [ERROR]
Info               [Exception: PATCH111
Err message: AutoUpgradePatchingException [OPatch failed to install ...]

Digging into the referenced log file revealed:

[Jun 25, 2025 10:48:51 AM] [WARNING]OUI-67200:Make failed to invoke "/usr/bin/make -f ins_rdbms.mk javavm_refresh ORACLE_HOME=/u01/app/oracle/product/dbhome_1927 OPATCH_SESSION=apply"....'Can't locate File/Copy.pm in @INC (you may need to install the File::Copy module) (@INC contains: /usr/local/lib64/perl5/5.32 /usr/local/share/perl5/5.32 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at /u01/app/oracle/product/dbhome_1927/javavm/install/update_javavm_binaries.pl line 64.
                                    BEGIN failed--compilation aborted at /u01/app/oracle/product/dbhome_1927/javavm/install/update_javavm_binaries.pl line 64.
                                    make: *** [ins_rdbms.mk:573: javavm_refresh] Error 2

The Workaround

You can avoid the problem by installing Perl before you start AutoUpgrade:

dnf -y install perl-core

Big shoutout to Martin Berger for providing a workaround.

Is My Server Affected

Use this command to test whether your server is affected:

perl -e 'use File::Copy;'

If it returns without any error or output, everything is fine. If you get an error, you must install Perl before proceeding:

perl -e 'use File::Copy;'
Can't locate File/Copy.pm in @INC (you may need to install the File::Copy module) (@INC contains: /usr/local/lib64/perl5/5.32 /usr/local/share/perl5/5.32 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at -e line 1.
BEGIN failed--compilation aborted at -e line 1.

Why Does It Happen

  • AutoUpgrade starts by installing the Oracle home and in some cases also applies the Release Update using the -applyRU command line parameter.

  • Next, it installs additional patches using OPatch. Some patches, like the OJVM bundle patch, requires a compilation using make. This process fails if there’s no usable Perl installation.

  • There are several MOS notes on this specific issues advising to set the PERL5LIB environment variable (Patching fails during relink , with error code 102 :: Fatal error: Command failed for target `javavm_refresh’ (Doc ID 2002334.1)).

  • To solve an unrelated issue, AutoUpgrade doesn’t always set PERL5LIB when calling OPatch and this leads to this error during the compilation.

  • We’re trying to find a smart way to handle this (without re-introducing the other issue).

To summarize, you get this error when:

  • Your operating system doesn’t have a normal Perl installation. You might have the perl executable, but not the normal modules.
  • You install the OJVM bundle patch.

The Rabbit Hole

I learned something new about Perl today. Do you dare walking down that rabbit hole with me? I’ll make it short.

  • The error from make mentions the missing Perl module, File::Copy:

    /usr/bin/make -f ins_rdbms.mk javavm_refresh ORACLE_HOME=/u01/app/oracle/product/dbhome_1927 OPATCH_SESSION=apply"....'Can't locate File/Copy.pm in @INC (you may need to install the File::Copy module)

  • A Perl module can be loaded from disk. Perl automatically attempts to find the file in the following locations:

    perl -e 'print "@INC\n"'
/usr/local/lib64/perl5/5.32
/usr/local/share/perl5/5.32
/usr/lib64/perl5/vendor_perl
/usr/share/perl5/vendor_perl
/usr/lib64/perl5
/usr/share/perl5

  • So, I can check these directories for the Perl module. To find the File::Copy module, I need to check in the File subdirectory for the file Copy.pm:

    ll /usr/local/lib64/perl5/5.32/File/Copy.pm
ll /usr/local/share/perl5/5.32/File/Copy.pm
ll /usr/lib64/perl5/vendor_perl/File/Copy.pm
ll /usr/share/perl5/vendor_perl/File/Copy.pm
ll /usr/lib64/perl5/File/Copy.pm
ll /usr/share/perl5/File/Copy.pm
ls: cannot access '/usr/local/lib64/perl5/5.32/File/Copy.pm': No such file or directory
ls: cannot access '/usr/local/share/perl5/5.32/File/Copy.pm': No such file or directory
ls: cannot access '/usr/lib64/perl5/vendor_perl/File/Copy.pm': No such file or directory
ls: cannot access '/usr/share/perl5/vendor_perl/File/Copy.pm': No such file or directory
ls: cannot access '/usr/lib64/perl5/File/Copy.pm': No such file or directory
ls: cannot access '/usr/share/perl5/File/Copy.pm': No such file or directory

  • The file is not found in any of the directories, so Perl will fail to load the module.

  • Had the file been found in just one location, all would be good.

Happy patching!

Where Is The OJVM Bundle Patch For Oracle Database 23ai?

A customer asked me:

Where do I download the OJVM bundle patch for Oracle Database 23ai? I can’t find it anywhere.

In Oracle Database 23ai, Oracle ships fully updated gold images instead of individual Release Updates. The customer installed the latest gold image and it didn’t contain the OJVM bundle patch:

$ORACLE_HOME/OPatch/opatch lspatches
37701424;OCW RELEASE UPDATE 23.8.0.25.04 (37701424) Gold Image
37701421;Database Release Update : 23.8.0.25.04 (37701421) Gold Image

In Oracle Database 19c, you would be able to see the OJVM bundle patch as a separate patch:

$ $ORACLE_HOME/OPatch/opatch lspatches
35648110;OJVM RELEASE UPDATE: 19.21.0.0.231017 (35648110)
35643107;Database Release Update : 19.21.0.0.231017 (35643107)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

Why is it missing from Oracle Database 23ai gold image and why can’t you download it separately?

OJVM In Oracle Database 23ai

Starting in Oracle Database 21c, there is no longer a separate OJVM bundle patch. Instead, the fixes are fully included in the Release Updates.

… the OJVM component patch has been incorporated within the 21c Database Release Update (RU) patch

Oracle Recommended Patches – “Oracle JavaVM Component Database PSU and Update” (OJVM PSU and OJVM Update) Patches (Doc ID 1929745.1)

Also, there is no longer any interuption or brownout when you start Datapatch. The OJVM fixes are now fully RAC Rolling Installable and Standby-First Installable.

So, just get the newest gold image in Oracle Database 23ai and that’s it.

What About Oracle Database 19c

Let’s just recap the situation when you have OJVM in Oracle Database 19c.

  • The patch is RAC Rolling Installable
  • Standby-First Installable (as long as you don’t have Active Data Guard and extensively use OJVM on the standby database)

First, are you using OJVM or can you remove it?

Then, how do you patch OJVM?

  1. Start by downloading the OJVM bundle patch.
    1. Use AutoUpgrade and include the OJVM keyword in your patch specification, like patch1.patch=OPATCH,RU,OJVM
    2. Use the Download Assistant on My Oracle Support
  2. When you start Datapatch and it applies the OJVM fixes, there is a short brownout while the Java system is reloaded. Any session that tries to use Java at that point will get ORA-29548 but will be able to retry after a few seconds.

Here’s a little video with additional information.

What About Grid Infrastructure?

You should not apply the OJVM bundle patch to your Grid Infrastructure homes.

OJVM PSU is not needed in the Grid home, only in the database home.

Oracle Recommended Patches – “Oracle JavaVM Component Database PSU and Update” (OJVM PSU and OJVM Update) Patches (Doc ID 1929745.1)

Happy patching!

Is Oracle Java Virtual Machine (OJVM) Used In My Database?

In Oracle Database, you can store and execute Java code directly in the database using Oracle Java Virtual Machine (OJVM). You can take your Java code, put it into the database and execute it. Now your code sits right next to the data, and you can get some amazing benefits.

But it comes at a price:

It’s really great if you are using OJVM; it has awesome functionality. But if you don’t, consider removing it. When I talk to customers, the big question is often:

How do I know if it is in use in my database?

Setting Things Straight

First, this blog post is about using OJVM: Java code in the database. Often, people mistakenly think they use OJVM because they have a Java application connecting to the database. They don’t. Having a Java application connecting to the database and using OJVM are two completely different things.

OJVM is often also referred to as:

  • JAVAVM
  • JServer JAVA Virtual Machine

Is OJVM Installed?

You can check if the OJVM component is installed in the database:

SQL> select con_id, comp_id, comp_name, version, status
     from cdb_registry
     where comp_id='JAVAVM';

If the query returns no rows, then OJVM is not installed.

Has Someone Added Java Code?

You can check if someone has added Java code to the database:

SQL> select con_id, owner, oracle_maintained, status, count(*) 
     from cdb_objects
     where object_type like '%JAVA%' 
     group by con_id, owner, oracle_maintained, status;

The column ORACLE_MAINTAINED indicates whether a regular user added it. If a user has added Java code, you can use the columns CREATED and LAST_DDL_TIME to find out when it happened. This might help you.

Is Someone Using OJVM Right Now ?

You can query x$kglob to determine the use of OJVM since the last startup. Refer to RAC Rolling Install Process for the "Oracle JavaVM Component Database PSU/RU" (OJVM PSU/RU) Patches (Doc ID 2217053.1) for details.

The MOS note also shows how to identify which sessions that use OJVM.

OJVM Dependencies

Be advised the following components in Oracle Database depend on OJVM. If you are using one of them, you can’t remove OJVM. The following components use OJVM:

  • Spatial Data Option (SDO)
  • Oracle XDK (XDK)
  • Oracle Multimedia (ORDIM)

Conclusion

Oracle Database is a converged database. You have so many great features directly available in the database. You can do many cool things with them – including OJVM. But if you are not using OJVM or any dependent components in Oracle Database, you can remove OJVM. It will save you time during patch installation and upgrades.

I used an example from oracle-base.com. Visit the article for all the details.

Appendix

Further Reading

Test Data

conn / as sysdba

--Query the current status
select comp_id, comp_name, version, status
from dba_registry
where comp_id='JAVAVM';

select con_id, owner, oracle_maintained, status, count(*) 
from cdb_objects
where object_type like '%JAVA%' 
group by con_id, owner, oracle_maintained, status;

--Shows status in current container and on current instance
--since last startup (MOS Doc ID 2217053.1)
select count(*) from x$kglob where KGLOBTYP = 29 OR KGLOBTYP = 56;

--Create user and small java code 
create tablespace appts;
create user appuser identified by appuser;
alter user appuser default tablespace appts;
grant dba to appuser;
conn appuser/appuser
--Thanks to oracle-base.com for a good example
--For further details:
--https://oracle-base.com/articles/8i/jserver-java-in-the-database
CREATE OR REPLACE AND COMPILE JAVA SOURCE NAMED "Mathematics" AS
import java.lang.*;
import java.sql.*;
import oracle.sql.*;
import oracle.jdbc.driver.*;

public class Mathematics
{
  
  public static int addNumbers (int Number1, int Number2)
  {
    try
    {
      int iReturn = -1;
      
      // Connect to the database
      Connection conn = null;
      OracleDriver ora = new OracleDriver(); 
      conn = ora.defaultConnection(); 
      
      // Check record exists, and create it if it doesn't
      Statement statement = conn.createStatement();
      ResultSet resultSet = statement.executeQuery("SELECT " + Number1 + " + " + Number2 + " FROM dual");
      if (resultSet.next())
      {
        iReturn = resultSet.getInt(1);
      }
      resultSet.close();
      statement.close();
      conn.close();

      return iReturn;
    }
    catch (Exception e)
    {
      return -1;
    }
  } 

};
/
show errors java source "Mathematics"

--Query the current status
conn / as sysdba
select con_id, owner, oracle_maintained, status, count(*) 
from cdb_objects
where object_type like '%JAVA%' 
group by con_id, owner, oracle_maintained, status;

select comp_id, comp_name, version, status
from dba_registry
where comp_id='JAVAVM';

select count(*) from x$kglob where KGLOBTYP = 29 OR KGLOBTYP = 56;

--Expose java code
conn appuser/appuser
CREATE OR REPLACE FUNCTION AddNumbers (p_number1  IN  NUMBER, p_number2  IN  NUMBER) RETURN NUMBER
AS LANGUAGE JAVA 
NAME 'Mathematics.addNumbers (int, int) return int';
/

--Query the current status
conn / as sysdba
select con_id, owner, oracle_maintained, status, count(*) 
from cdb_objects
where object_type like '%JAVA%' 
group by con_id, owner, oracle_maintained, status;

select comp_id, comp_name, version, status
from dba_registry
where comp_id='JAVAVM';

select count(*) from x$kglob where KGLOBTYP = 29 OR KGLOBTYP = 56;

--Use Java code
conn appuser/appuser
SELECT addNumbers(1,2)
FROM   dual;

--Check which sessions have actively used Java since last startup
--Refer to MOS Doc ID 2217053.1 for the query

Can I Run Datapatch When Users Are Connected

The short answer is: Yes! The longer answer is: Yes, but very busy systems or in certain situations, you might experience a few hiccups.

The obvious place to look for the answer would be in the documentation. Unfortunately, there is no Patching Guide similar to the Upgrade Guide. The information in this blog post is pieced together from many different sources.

A few facts about patching with Datapatch:

  • The database must be open in read write mode.
  • You can’t run Datapatch on a physical standby database – even if it’s open (Active Data Guard).
  • A patch is not fully installed until you have executed Datapatch successfully.

How To

First, let me state that it is fully supported to run Datapatch on a running database with users connected.

The procedure:

  1. Install a new Oracle Home and use OPatch to apply the desired patches.
  2. Shut down the database.
  3. Restart the database in the new, patched Oracle Home.
  4. Downtime is over! Users are allowed to connect to the database
  5. Execute ./datapatch -verbose.
  6. End of procedure. The patch is now fully applied.

Often users move step 4 (Downtime is over) to the end of the procedure. That’s of course also perfectly fine, but it does extend the downtime needed and often is not needed.

What About RAC and Data Guard

The above procedure is exactly what happens in a rolling patch apply on a RAC database. When you perform a rolling patch apply on a RAC database, there is no downtime at all. You use opatchauto to patch a RAC database. opatchauto restarts all instances of the database in the patched Oracle Home in a rolling manner. Finally, it executes datapatch on the last node. Individual instances are down temporarily, but the database is always up.

It is a similar situation when you use the Standby First Patch Apply. First, you restart all standby databases in the patched Oracle Home. Then, you perform a switchover and restart the former primary database in the patched Oracle Home. Finally, you execute datapatch to complete the patch installation. You must execute datapatch on the primary database.

Either way, don’t use Datapatch until all databases or instances run on the new, patched Oracle Home.

That’s It?

Yes, but I did write initally that there might be hiccups.

Waits

Datapatch connects to the database like any other session to make changes inside the database. These changes could be:

  • Creating new tables
  • Altering existing tables
  • Creating or altering views
  • Recreating PL/SQL packages like DBMS_STATS

Imagine this scenario:

  1. You restart the database in the patched Oracle Home.
  2. A user connects and starts to use DBMS_STATS.
  3. You execute datapatch.
    1. Datapatch must replace DBMS_STATS to fix a bug.
    2. Datapatch executes CREATE OR REPLACE PACKAGE SYS.DBMS_STATS .....
    3. The Datapatch database session go into a wait.
  4. User is done with DBMS_STATS.
  5. The Datapatch session come out of wait and replace the package.

In this scenario, the patching procedure was prolonged due to the wait. But it completed eventually.

Hangs

From time to time, we are told that Datapatch hangs. Most likely, it is not a real hang, but just a wait on a lock. You can identify the blocking session by using How to Analyze Library Cache Timeout with Associated: ORA-04021 ‘timeout occurred while waiting to lock object %s%s%s%s%s.’ Errors (Doc ID 1486712.1).

You might even want to kill the blocking session to allow Datapatch to do its work.

Timeouts

What will happen in the above scenario if the user never releases the lock on DBMS_STATS? By default, Datapatch waits for 15 minutes (controlled by _kgl_time_to_wait_for_locks) before throwing an error:

ORA-04021: timeout occurred while waiting to lock object

To resolve this problem, restart Datapatch and ensure that there are no blocking sessions. Optionally, increase the DDL timeout:

./datapatch -ddl_lock_timeout <time-in-sec>

Really Busy Databases

I recommend patching at off-peak hours to reduce the likelihood of hitting the above problems.

If possible, you can also limit the activity in the database while you perform the patching. If your application is using e.g. DBMS_STATS and locking on that object is often a problem, you can hold off these sessions for a little while.

The Usual Suspects

Based on my experience, when there is a locking situation, these are often the sinner:

  • Scheduler Jobs – if you have jobs runnings very frequently, they may all try to start when you restart your database in the new Oracle Home. Suspend the workload temporarilty by setting job_queue_processes to 0.
  • Advanced Queeing – if you have lots of activities happening via AQ, you can suspend it temporarily by setting aq_tm_processes to 0. If you disable the scheduler, you also disable AQ.
  • Materialized Views – when the database refreshes materialized views it uses internal functionality (or depending objects) that Datapatch needs to replace. By disabling the scheduler, you also disable the materialized view refreshes.
  • Backup jobs – I have seen several situations where Datapatch couldn’t replace the package dbms_backup_restore because the backup system took archive backups.

Last Resort

If you want to be absolutely sure no one intervenes with your patching, use this approach. But it means downtime:

  1. SQL> startup restrict
  2. ./datapatch -verbose
  3. SQL> alter system disable restricted session;

I don’t recommend starting in upgrade mode. To get out of upgrade mode a database restart is needed extending the downtime window.

Datapatch And Resources

How much resources does Datapatch need? Should I be worried about Datapatch depleting the system?

No, you should not. The changes that Datapatch needs to make are not resource-intensive. However, a consequence of the DDL statements might be object invalidation. But even here, you should not worry. Datapatch will automatically recompile any ORACLE_MAINTAINED object that was invalidated by the patch apply. But the recompilation happens serially, i.e., less resources needed.

Of course, if you system is running at 99% capacity, it might be a problem. On the other hand, if your system is at 99%, patching problems are probably the least of your worries.

What About OJVM

If you are using OJVM and you apply the OJVM bundle patch, things are a little different.

Release RAC Rolling Standby-First Datapatch
Oracle Database 21c Fully No No Datapatch downtime.
Oracle Database 19c + 18c Partial No No Datapatch downtime, but java system is patched which requires ~10 second outage. Connected clients using java will receive ORA-29548.
Oracle Database 12.2 + 12.1 No No Datapatch must execute in upgrade mode.
Oracle Database 11.2.0.4 No No Similar to 12.2 and 12.1 except you don’t use Datapatch.

Mike Dietrich also has a good blog that you might want to read: Do you need STARTUP UPGRADE for OJVM?

What About Oracle GoldenGate

You should stop Oracle GoldenGate when you execute datapatch. When datapatch is done, you can restart Oracle GoldenGate.

If you are manually recompiling objects after datapatch, I recommend that you restart Oracle GoldenGate after the recompilation.

The above applies even if the patches being applied does not contain any Oracle GoldenGate specific patches.

Oracle GoldenGate uses several objects owned by SYS. When datapatch is running it might change some of those objects. In that case, unexpected errors might occur.

Recommendations

Based on my experience, these are my recommendations

Before Patching

  • Recompile invalid objects (utlrp).
  • Perform a Datapatch sanity check ($ORACLE_HOME/OPatch/datapatch -sanity_checks).
  • Postpone your backup jobs.
  • Stop any Oracle GoldenGate processes that connects to the database.
  • Disable the scheduler.

Patching

  • Always use the latest OPatch.
  • Always use out-of-place patching, even for RAC databases.
  • Always enable verbose output in Datapatch ($ORACLE_HOME/OPatch/datapatch -verbose).

After Patching

  • If applicable, re-enable
    • Backup jobs.
    • Oracle GoldenGate processes.
    • The scheduler.
  • Check Datapatch output. If Datapatch failed to recompile any objects, a message is printed to the console. If you patch interactively, you can find the same information in the log files.

Still Don’t Believe Me?

In Autonomous Database (ADB), there is no downtime for patching. An ADB runs on RAC and patching is fully rolling. The automation tooling executes Datapatch while users are connected to the database.

Of course, one might run into the same issues described above. But Oracle have automation to handle the situation. If necessary, the database kills any sessions blocking Datapatch. In the defined maintenance window in your ADB, you may end up in a situation that a long-running, blocking session terminates because it was blocking a Datapatch execution. But if you minimize your activities in the defined maintenance windows, then chances of that happening is minimal.

Conclusion

Go ahead and patch your database with Datapatch while users are connected.

Further Reading