Upgrade Oracle Database 11g to 19c Using RMAN Incremental Backups

A customer wanted help upgrading and moving to new hardware:

I have an 11g database running on old hardware. I want to upgrade to 19c and move to new hardware. I can’t use Data Guard to move the database because I can’t install 11g binaries on the new hardware. What do I do?

Data Guard would be the obvious choice, but wasn’t possible here. But we can use RMAN incremental backups.

This allows us to move the database with little downtime and then perform the upgrade.

The Plan

Here’s an overview of the plan:

  1. Perform a level 0 backup of the 11g database.
  2. Restore the backup into a 19c instance on the new hardware.
  3. Prepare the database for upgrade.
  4. Perform a level 1 backup.
  5. Recover the database.
  6. Upgrade the database.

We can perform steps 1-2 in advance and save time during the outage.

Any newer version of Oracle AI Database can restore backups from a previous release. But to upgrade the database, we must be within the limitations for a direct upgrade. Oracle Database 19c supports direct upgrades from 11g.

Step 1: Initial Backup

The database is called UPGR and runs on the old hardware as an 11g database.

  1. On the old system, I start by doing a level 0 backup of the source database:
    RMAN> CONNECT TARGET /
          RUN {
          ALLOCATE CHANNEL c1 DEVICE TYPE DISK;
          ALLOCATE CHANNEL c2 DEVICE TYPE DISK;
    
          BACKUP AS COMPRESSED BACKUPSET
             INCREMENTAL LEVEL 0
             DATABASE
             FORMAT '/u01/app/oracle/backup/L0_%d_%T_%U.bkp'
             PLUS ARCHIVELOG
             FORMAT '/u01/app/oracle/backup/L0_%d_%T_%U.bkp'
             TAG 'LEVEL0';
    
          BACKUP CURRENT CONTROLFILE
             FORMAT '/u01/app/oracle/backup/CTL.bkp'
             TAG 'CONTROLFILE';
    
          RELEASE CHANNEL c1;
          RELEASE CHANNEL c2;
          }
    
    • I store the backups on an NFS share that’s accessible to the new system as well.
    • I’m doing a compressed backup. Be sure you’re licensed for that.
    • You can customize the backup script.
  2. I create a PFile:
    SQL> CREATE PFILE='/u01/app/oracle/backup/pfile.txt'
         FROM SPFILE;
    

Step 2: Configure Instance and Restore

I create a new instance on the new hardware. I must use the same name, UPGR.

  1. On the new hardware, I create a PFile for my target instance. I use the original PFile as a template and make the necessary changes. Here’s the PFile that I’ll use:
    *._cursor_obsolete_threshold=1024
    *.audit_file_dest='/u01/app/oracle/admin/UPGR/adump'
    *.audit_trail='db'
    *.compatible='11.2.0.4.0'
    *.control_files='/u02/oradata/UPGR/controlfile/control01.ctl'
    *.db_block_size=8192
    *.db_create_file_dest='/u02/oradata'
    *.db_name='UPGR'
    *.db_recovery_file_dest_size=12884901888
    *.db_recovery_file_dest='/u02/fast_recovery_area'
    *.diagnostic_dest='/u01/app/oracle'
    *.pga_aggregate_target=1G
    *.sga_target=4G
    
    • I keep the same db_name.
    • Adjust locations, like db_create_file_dest and control_files accordingly.
  2. I create the directory specified by audit_file_dest:
    mkdir -p /u01/app/oracle/admin/UPGR/adump
    
  3. I set the environment and create a new password file:
    export ORACLE_HOME=/u01/app/oracle/product/19
    export ORACLE_BASE=/u01/app/oracle
    export ORACLE_SID=UPGR
    export PATH=$ORACLE_HOME/bin:$PATH
    
    $ORACLE_HOME/bin/orapwd file=orapwUPGR password=MyS3cr3tPassw0rd!
    
  4. Next, I start the new instance in NOMOUNT mode:
    SQL> STARTUP NOMOUNT
    
  5. I restore the control file:
    RMAN> CONNECT TARGET /
          RUN {
          RESTORE CONTROLFILE
             FROM '/u01/app/oracle/backup/CTL.bkp';
          }
    
    • I restore from the NFS share.
  6. Then, I mount the database and start the restore:
    RMAN> CONNECT TARGET /
          ALTER DATABASE MOUNT;
          RUN {
          ALLOCATE CHANNEL c1 DEVICE TYPE DISK;
          ALLOCATE CHANNEL c2 DEVICE TYPE DISK;
    
          SET NEWNAME FOR DATABASE TO NEW;
    
          CATALOG START WITH '/u01/app/oracle/backup/' NOPROMPT;
    
          RESTORE DATABASE;
          SWITCH DATAFILE ALL;
          RECOVER DATABASE UNTIL AVAILABLE REDO;
    
          RELEASE CHANNEL c1;
          RELEASE CHANNEL c2;
          }
    
    • I use Oracle Managed Files (OMF), so I use SET NEWNAME accordingly.
    • I use CATALOG to tell RMAN where to find the backups.
    • I restore the database and recover until there’s no more redo.

Step 3: Prepare for Upgrade

Downtime starts now.

  1. On the source system, I create an AutoUpgrade config file, so I can check the upgrade readiness of my database.
    global.global_log_dir=/home/oracle/autoupgrade-logs
    upg1.source_home=/u01/app/oracle/product/11.2.0.4
    upg1.target_home=/tmp
    upg1.target_version=19
    upg1.sid=UPGR
    
    • I can’t set target_home to the real value because it doesn’t exist on the source server. I use a fake entry instead.
    • AutoUpgrade normally deduces the target_version from target_home, but since it doesn’t exist, I need to specify it manually.
  2. I perform the preupgrade check:
    java -jar autoupgrade.jar -config UPGR.cfg -mode analyze
    
    • Check the preupgrade summary report.
    • I use the latest version of AutoUpgrade.
  3. I run the preupgrade fixups:
    java -jar autoupgrade.jar -config UPGR.cfg -mode fixups
    
  4. Then, I run an incremental backup:
    RMAN> CONNECT TARGET /
          RUN {
          ALLOCATE CHANNEL c1 DEVICE TYPE DISK;
          ALLOCATE CHANNEL c2 DEVICE TYPE DISK;
    
          BACKUP AS COMPRESSED BACKUPSET
             INCREMENTAL LEVEL 1
             DATABASE
             FORMAT '/u01/app/oracle/backup/L1_%d_%T_%U.bkp'
             PLUS ARCHIVELOG
             FORMAT '/u01/app/oracle/backup/L1_%d_%T_%U.bkp'
             TAG 'LEVEL1';
    
          RELEASE CHANNEL c1;
          RELEASE CHANNEL c2;
          }
    
    • The incremental backup captures the last changes to my database.
  5. I shut down the database on the old system.

Step 4: Recover

  1. On the target database, I recover the incremental backup.
    RMAN> CONNECT TARGET /
          RUN {
          ALLOCATE CHANNEL c1 DEVICE TYPE DISK;
          ALLOCATE CHANNEL c2 DEVICE TYPE DISK;
    
          CATALOG START WITH '/u01/app/oracle/backup/' NOPROMPT;
    
          RECOVER DATABASE UNTIL AVAILABLE REDO;
    
          RELEASE CHANNEL c1;
          RELEASE CHANNEL c2;
          }
    
    • I use the CATALOG command to tell RMAN about the new backup.
    • I recover the database until there is no more redo.
  2. Then, I open the database in upgrade mode:
    SQL> ALTER DATABASE OPEN RESETLOGS UPGRADE;
    
    • I use the RESETLOGS clause to create redo logs.
    • I must open in upgrade mode to start the upgrade. If I try to open in normal mode, I’ll get an error ORA-00704: bootstrap process failure.

Step 5: Upgrade

The database is open in upgrade mode. I must perform the upgrade to 19c.

  1. On the target system, I create an AutoUpgrade config file:
    global.global_log_dir=/home/oracle/autoupgrade-logs
    upg1.source_home=/tmp
    upg1.target_home=/u01/app/oracle/product/19
    upg1.sid=UPGR
    
    • I can’t set source_home to the real value because it doesn’t exist on the target server. I use a fake entry instead.
  2. I perform the upgrade:
    java -jar autoupgrade.jar -config UPGR.cfg -mode upgrade
    
    • I start with -mode upgrade to complete the upgrade. Don’t use -mode deploy.
    • I use the latest version of AutoUpgrade.
  3. After a while, the upgrade completes. All done.

That’s it

RMAN incremental backup is a great feature to reduce downtime when Data Guard is not an option. You can restore to a higher release of Oracle AI Database as long as you stay within the limits for a direct upgrade.

You can enhance the procedure:

  • Run additional level 1 incremental backup/restores to minimize the time it takes to perform the last one.
  • Enable Block Change Tracking on the old database to shorten the time it takes to do incremental backups.
  • Run an additional AutoUpgrade preupgrade check days in advance to get early notice on any showstoppers.

Happy upgrading!

AutoUpgrade New Features: Upgrade RMAN Catalog Schema

With the latest version, 24.8, AutoUpgrade can upgrade the RMAN catalog schema after patching and upgrading. This is useful to those who take RMAN backups and duplicate their RMAN metadata to a catalog database.

If you don’t upgrade the catalog schema after patching and upgrading, you’ll see this message in the RMAN output:

PL/SQL package RCO.DBMS_RCVCAT version 19.24.00.00. in RCVCAT database is not current
PL/SQL package RCO.DBMS_RCVMAN version 19.24.00.00 in RCVCAT database is not current

Details

  • After patching or upgrading, AutoUpgrade upgrades the RMAN catalog schema in the postupgrade stage.
  • AutoUpgrade connects with RMAN to the recovery catalog and issues the upgrade catalog command.
  • AutoUpgrade does not execute dbmsrmansys.sql. Normally, this is only needed for the upgrade of the first catalog schema of a given release (like for the first database on Oracle Database 23ai), and even then, it might not be needed.

How To

  • Specify the connect string to the catalog database in the AutoUpgrade config file:

    <prefix>.rman_catalog_connect_string=catalogdb
    
    • catalogdb is a TNS alias to the catalog database.
  • Start AutoUpgrade to load the username and password for the recovery catalog:

    java -jar autoupgrade.jar -config ... -load_password
    
  • Switch to the password group RMAN:

    group rman
    
  • Add the username and password for a specific database:

    add <ORACLE_SID> -user <catalog_schema_name>
    
    • AutoUpgrade prompts for the password
  • Save the changes and exit the load password console.

    save
    exit
    
  • Start AutoUpgrade in deploy mode:

    java -jar autoupgrade.jar -config ... -mode deploy
    

Happy Upgrading, Happy Patching

  • You can enhance the solution using an after_action script that starts a level 1 backup after the job. The after_action script takes place after the postupgrade stage, where AutoUpgrade upgrades the catalog schema.

  • Version 24.8 of AutoUpgrade does not support this feature when you use the -patch command line option. This is coming in a later version.

Appendix

Further Reading

Invalid credentials

  • When you enter the catalog credentials into the AutoUpgrade keystore, AutoUpgrade validates the credentials. Any errors result in AutoUpgrade returning the following message:

    Invalid credentials, please try again.
    
  • To debug, run the following command:

    $ORACLE_HOME/bin/rman TARGET / rcvcat <catalog_user>@<rman_catalog_connect_string>
    
  • Check the log files in:

    <global_log_dir>/log/cfgtoollogs/upgrade/auto
    

What to Do about RMAN Recovery Catalog When You Upgrade Oracle Database

When you upgrade your Oracle Database, you want to ensure your backup strategy is not compromised. The RMAN recovery catalog is a key part of your backup strategy. What do you need to take care of when upgrading Oracle Database? Don’t let the upgrade of your Oracle Database jeopardize your RMAN backup strategy.

First, let’s agree on the terminology:

  • Target Database – The database that you want to backup. There is where your data is stored.
  • Catalog Database – A regular Oracle Database with one or more catalog schemas.
  • Catalog Schema – A schema inside the catalog database which holds the metadata about the RMAN backups. You can register multiple databases in the same catalog schema.
  • Recovery Catalog – The catalog schema and catalog database together form the recovery catalog.

The topology of RMAN catalog and Oracle Database

What’s Required?

RMAN Client

The RMAN client you use for the backup must be the same version as the target database. I find it easiest to always use the RMAN executable from the same Oracle Home as the target database.

Previously, this was not a requirement, but it is in current versions.

Catalog Schema

The catalog schema version must be the same or higher compared to the RMAN client. When you upgrade the target database and start to use a newer RMAN client, you also need to upgrade the catalog schema.

  • Upgrade the target database.
  • Start RMAN using the executable from the target database Oracle Home.
  • Connect to the target database and recovery catalog and perform the upgrade:
    $ $ORACLE_HOME/bin/rman
    
    RMAN> connect target /
    RMAN> connect catalog <catalog_schema>/<catalog_schema_password>@<catalog_database_alias>
    RMAN> upgrade catalog noprompt;
    

If you register multiple databases into the same catalog schema, and you have already upgraded another target database to the same version, then there is no need to upgrade the catalog schema again.

When you upgrade with AutoUpgrade, I find it easiest to allow AutoUpgrade to perform the upgrade for me.

Since the catalog schema is backward compatible, you can downgrade the target database and still use a higher version catalog schema. In case of a target database downgrade, no changes are needed in the catalog schema.

Catalog Database

As long as the catalog database runs on a supported version, you should be home safe. Target databases on Oracle Database 19c support using catalog databases all the way back to 10.2.0.3. Hopefully, you don’t use such old versions anymore. The version of the catalog database does not have to match either the catalog schema or the target database.

You can upgrade the catalog database like any other database. Use AutoUpgrade in deploy mode, and that’s it. A catalog database requires no special attention. But you can always run $ORACLE_HOME/rdbms/admin/dbmsrmansys.sql in the catalog database to ensure all the prerequisites are met (plus dbmsrmanvpc.sql for VPC users).

If your catalog database is on Oracle Database 11g, there are a few details in My Oracle Support Doc ID 1970049.1 to be aware of.

In some situations, building a new catalog database is desirable. Like:

  • Your catalog database is on Standard Edition. Nowadays, a catalog database must be Enterprise Edition.
  • Your catalog database is very old and can’t be directly upgraded.

The IMPORT CATALOG command can move a recovery catalog or selected catalog schemas into a new catalog database.

Example

Given the above requirements, here is an example:

  • You have three target databases running on various releases, let’s say Oracle Database 19c, 12.1.0.2, and 11.2.0.4.
  • Those target databases must use a catalog schema matching their respective release. You have three catalog schemas in total:
    • One catalog schema is on catalog schema version 19
    • One catalog schema is on catalog schema version 12.1.0.2
    • One catalog schema is on catalog schema version 11.2.0.4
  • You have just one catalog database. The catalog database is on Oracle Database 21c.
  • You can store all three catalog schemas in the same catalog database. The catalog schema version and the catalog database release do not have to match.

Now imagine you upgrade the 11.2.0.4 database to 12.2.0.1. Now, you must upgrade the catalog schema version to 12.2.0.1 using the UPGRADE CATALOG command. Also, you must switch to an RMAN client on 12.2.0.1.

What About Zero Data Loss Recovery Appliance?

If you are fortunate to have a (ZDLRA), you must not touch the catalog database yourself. All operations on the catalog database must happen via racli. When you update the appliance software, it will upgrade the catalog database underneath the hood.

You can still upgrade the catalog schema using RMAN and the upgrade catalog command.

My Recommendation

  1. Keep your catalog database on Long-Term Support releases. At the time of writing, it means keeping your catalog database on Oracle Database 19c.
  2. Upgrade your catalog database to the next Long-Term Support release before you upgrade the target databases.
  3. Apply Release Updates regularly.

Thanks

Thanks to my good colleagues, Jony and Andrew, for their help and good pointers. Much appreciated.

Further Reading