How to Trace Oracle Data Pump

If you ever encounter problems with Oracle Data Pump, you can use this recipe to get valuable tracing.

Over the years, I’ve helped many customers with Data Pump issues. The more information you have about a problem, the sooner you can come up with a solution. Here’s my list of things to collect when tracing a Data Pump issue.

Daniel’s Tracing Recipe

1. AWR

  • Be sure you have a proper license to use AWR.

  • Set the snapshot interval to 15 minutes and create a new snapshot:

    exec dbms_workload_repository.modify_snapshot_settings(null, 15);
    exec dbms_workload_repository.create_snapshot;
    
  • If you are on Multitenant, do so in the root container and in the PDB.

2. SQL Trace

  • Depending on the nature of the problem, you can enable SQL trace of the Data Pump processes:

    alter system set events 'sql_trace {process: pname = dw | process: pname = dm} level=8';
    
    • You can change the trace level as required.
    • dm is the Data Pump control process, dw are worker processes.
  • If you already know the SQL ID causing problems, you can enable tracing for just that SQL:

    alter system set events 'sql_trace[SQL: <sql-id>]';
    
    • Replace <sql-id> with the offending SQL ID.

3. Start Data Pump

  • Start the Data Pump job that you want to trace:
    expdp ... metrics=yes logtime=all trace=<trace-setting>
    impdp ... metrics=yes logtime=all trace=<trace-setting>
    

4. AWR

  • Be sure you have a proper license to use AWR.

  • When the Data Pump job completes or after you stopped it, reset the snapshot interval to the original value and create a new AWR snapshot:

    exec dbms_workload_repository.modify_snapshot_settings(null, <original-value>);
    exec dbms_workload_repository.create_snapshot;
    
  • Create an AWR report spanning the entire period:

    @?/rdbms/admin/awrrpt
    
    • If needed, you can later on create AWR reports for a shorter period.
  • If you are on Multitenant, do so in the root container and in the PDB.

Get the Information

Collect the following information:

  1. The Data Pump log file.
  2. AWR reports – on CDB and PDB level
  3. Data Pump trace files
    • Stored in the database trace directory
    • Control process file name: *dm*
    • Worker process file names: *dw*

This should be a great starting point for diagnosing your Data Pump problem.

What Else

  • Remember, you can use the Data Pump Log Analyzer to quickly generate an overview and to dig into the details.

  • Regarding Data Pump parameters metrics=yes and logtime=all. You should always have those in your Data Pump jobs. They add very useful information at no extra cost. In Oracle, we are discussing whether these should be default in a coming version of Data Pump.

Leave a comment and let me know your favorite way of tracing Oracle Data Pump.

Our Real World Database Upgrade and Migration Workshop Is Back on the Road

Now that Oracle CloudWorld 2024 is over, we have time to spare, so it is time to re-ignite our full-day workshop:

Real World Database Upgrade and Migration 19c and 23ai

Next stops on our tour:

Workshops coming to Berlin, Zurich, and Oslo

Click on the city name to sign up – for free! Save your seat before the workshop fills up.

Mike Dietrich, Rodrigo Jorge, and I will present in English. It is an in-person event only.

What Is It?

It is your chance to meet with our product management team for a full day:

  • How to take full advantage of the new features and options in Oracle Database 19c and 23ai
  • The smoothest and fully unattended migration to the CDB architecture
  • Real World Best Practices and Customer Cases
  • Database and Grid Infrastructure Patching Best Practices
  • Performance Stability Prescription and Tips
  • The coolest new features in Oracle Database 23ai for DBAs and Developers

From a previous workshop

I hope to see you there.

All tech, no marketing!

Sure, Let Me Analyze This 200.000-Line Log File

Imagine importing a large database using Oracle Data Pump. In the end, Data Pump tells you success/failure and the number of errors/warnings encountered. You decide to have a look at the log file. How big is it?

$ du -h import.log
 29M   import.log

29 MB! How many lines?

$ wc -l import.log
  189931 import.log

Almost 200.000 lines!

How on earth can you digest that information and determine whether you can safely ignore the errors/warnings recorded by Data Pump?

Data Pump Logfile Analyzer

This is where Data Pump Logfile Analyzer (DPLA) can help you.

DPLA can summarize the log file into a simple report. Summary of a Data Pump job

It can give you an overview of each type of error. Showing the errors reported in a Data Pump log file

It can tell you where Data Pump spent the most time. Showing which Data Pump phases took the longest

It can produce an interactive HTML report. HTML report from Data Pump Log Analyzer

And so much more. It’s a valuable companion when you use Oracle Data Pump.

Tell Me More

DPLA is not an official Oracle tool.

It is a tool created by Marcus Doeringer. Marcus works for Oracle and is one of our migration superstars. He’s been involved in the biggest and most complicated migrations and knows the pain of digesting a 200.000-line log file.

He decided to create a tool to assist in the analysis of Data Pump log files. He made it available for free on his GitHub repo.

Give It a Try

Next time you have a Data Pump log file, try to use the tool. It’s easy, and instructions come with good examples.

If you like it, be sure to star his repo. ⭐

If you can make it better, I’m sure Marcus would appreciate a pull request.

Thanks, Marcus, good job! 💪

Oracle CloudWorld 2024 – It’s a Wrap

I can’t believe Oracle CloudWorld is already over. Although it has been very intense, it feels like it has just started. I love being amongst our customers and helping them use the Oracle Database in the best possible way.

Oracle CloudWorld banner

I still feel the thrill from the conference, but I know that post-conference blues are soon kicking in.

Slides

I encourage you to look at the slides from our presentations. We did present some new cool enhancements.

The audience at our patching session

Try Our Hands-On Labs

This year, we launched two brand-new hands-on labs:

You can try them as well in Oracle LiveLabs – FOR FREE!

Thanks

Thanks to you – our valued customer – for coming to our conference and engaging with us.

Thanks to my team: Mike, Rodrigo and Alex. All the content we deliver is a genuine team effort.

What’s Next

Stay tuned for more information about Oracle CloudWorld Tour. In early 2025, we will bring Oracle CloudWorld to your neighborhood.

Oracle Redbull racer on display

I hope to see you next year at Oracle CloudWorld, 13-16 October 2025.

Save $100 by subscribing to updates about CloudWorld 2025.

Oracle CloudWorld Day 3+4

Day 3 is the day of The Party. This year, with legendary The Journey playing.

CloudWorld The Party

Besides the music, all kinds of fun went on.

CloudWorld The Party

The Sphere

I had a few hours before The Party, so I went to catch a show at The Sphere. It’s an amazing event venue – the exterior is almost as impressing as the interior.

The Sphere in Las Vegas

I saw the most amazing U2 concert – recorded specially for the huge indoor screen. There is literally screens all over the place, so it’s like being right in the middle of it.

Next time you come to Oracle CloudWorld, be sure to book a show.

Announcements

In our Data Pump talk, we had several new features to present:

  • Faster creation of indexes
  • Instantly adding new constraints in NOVALIDATE mode

From the smiling faces in the audience, those new features were really well received. Stay tuned for more information.

Oracle Data Pump 23ai talk

The Beast

I also had the chance to present a story from the trenches.

Migrating The Beast

Migrating the beast

An interesting case about a 180 TB database generating 15 TB/day – moving from SPARC Solaris to Exadata. A great achievement by our friends from Entain.

Over the last years I’ve learned so much from working with this project. Never hesitate to reach out to a product manager when you have big projects ahead of you.

We love these kinds of projects!

Oracle CloudWorld Day 2

The big show started at Oracle CloudWorld 2024. The keynotes happened, and Larry announced Oracle Database@AWS.

The next important announcement: AutoUpgrade Patching is now ready with superpowers. One command and it will:

  • Download the patches from My Oracle Support.
  • Build a new Oracle home.
  • Patch your Oracle Database.

You don’t even need to specify patch numbers; just tell us you want the latest patches. Patching has never been easier.

AutoUpgrade patching makes it even easier

CloudWorld Hub

The CloudWorld hub is open, and there’s plenty to do:

  1. Check out the F1 racer and try one of the race simulators. It’s right behind the database demo booths.
  2. Relax with pinball, table football, and Guitar Hero. The pinball machine does bring back a lot of childhood memories. It’s right after the entrance on the right.
  3. Get an AI generated avatar for your profile pictures. The machine takes a picture and comes back with an AI generated picture of you on the Premiere League football pitch, F1 racing court, cricket game or sailboat.
  4. Get a bunch of stickers from all around and generate your own sticker at the database demo booth.
  5. Pet the dogs. Relax yourself by playing with some of the dogs. Yup, that’s right. There are dogs to give you a good amount of paw-love. Plus, the PS Websolutions booth has Kirby the Chihuahua, who’s also looking for someone to pet him.
  6. Claim your fair share of the swag.

Cool stuff at the hub

Enjoy!

Oracle CloudWorld Day 1

That’s the end of the first day of Oracle CloudWorld 2024.

My team and I spent the day in a full-day workshop about upgrade to Oracle Database 23ai and migration to the multitenant architecture.

Oracle CloudWorld is on

Pro Tips

Today was a kind of warm-up for tomorrow. Get the most out of the conference with these pro tips:

  1. You get the best coffee by far at Dandelion Chocolate near the front desk. Second best is at Zeppola Cafe at Sct. Marks Square.
  2. Get the Oracle Events app. It’s your trusted companion.
  3. Pick up your badge in good time. The lines for badge pick-up on Tuesday morning will be long.
  4. Bring a light sweater or jacket. The A/C is brutal in some places.
  5. Show up for the sessions in good time. Although you registered for a session, they let people in on a first-come-first-served basis. Don’t miss a seat by showing up late.
  6. Rate the sessions and leave feedback. We want to improve and ensure you get the most out of it.
  7. If you signed up for a hands-on lab or tutorial, be sure to bring your laptop. We do have loaners, but not everyone likes a US keyboard.
  8. Come to the CloudWorld hub and visit the demo booths. Product Managers from all areas are present and they are eager to answers all your questions. Engage with us, please.
  9. While at the CloudWorld hub, stop by the ACE lounge and ask one of the ACEs why you should become the next Oracle ACE. You’ll not regret joining the community. And ask when the chocolate tasting takes place.
  10. Stop by the merchandise shop and arm yourself with cool accessories.
  11. Take the 2024 Database Developer Survey. Swing by the Swag booth to claim your prize.

What’s your favorite pro tip? Leave a comment and enhance everyone’s CloudWorld experience.

Enjoy!

A Few Details about Using Refreshable Clone PDB for Non-CDB to PDB Migration

Our team has been advocating the use of refreshable clone PDB for non-CDB to PDB migrations using AutoUpgrade. It is a great feature and our entire team loves it – so does many of the customers we work with.

However, in a recent non-CDB to PDB migration, we encountered some issues with refreshable clone PDB and AutoUpgrade.

Can My Target Container Database Be a RAC Database?

Yes, this works perfectly fine.

Be aware that CREATE PLUGGABLE DATABASE statement scales out on all nodes in your cluster. By default, the database also uses parallel processes, so potentially, this will put quite a load on the source non-CDB. Consider restricting the use of parallel processes using the AutoUpgrade config file parameter:

upg1.parallel_pdb_creation_clause=4

Since the creation scales out on all nodes, all nodes must be able to resolve the connect identifier to the source non-CDB. If you use an alias from tnsnames.ora, be sure to add that on all nodes. Failure to do so will lead to an error during the CREATE PLUGGABLE DATABASE command:

ERROR at line 1:
ORA-65169: error encountered while attempting to copy file
+DATAC1/SRCDB/DATAFILE/system.262.1178083869
ORA-17627: ORA-12154: TNS:could not resolve the connect identifier specified
ORA-17629: Cannot connect to the remote database server

What Happens If the Source Database Extends a Data File?

If the source database extends a data file – either through AUTOEXTEND ON NEXT or manually by a user – the target database extends the matching data file as well. Here is an extract from the target alert log when it extends a data file:

2024-08-27T07:01:26.671975+00:00
PDB1(4):Media Recovery Log +RECOC1/SRCDB/partial_archivelog/2024_08_27/thread_2_seq_4.276.1178089277
2024-08-27T07:01:32.773191+00:00
PDB1(4):Resize operation completed for file# 26, fname +DATA/TGTCDB_HBZ_FRA/20A568D1FD5DB0A6E0633D01000AC89B/DATAFILE/srctbs02.290.1178089287, old size 10240K, new size 1058816K

It works with smallfile and bigfile tablespaces.

What Happens If I Create a Tablespace on the Source Database?

The target database attempts to create the same tablespace.

For this to work, one of the following must be true:

If either one of the above isn’t true, you’ll receive an error during ALTER PLUGGABLE DATABASE ... REFRESH:

ORA-00283: recovery session canceled due to errors
ORA-01274: cannot add data file that was originally created as
'+DATAC1/SRCDB/DATAFILE/srctbs04.282.1178091655'
You can use PDB_FILE_NAME_CONVERT instead.

It works with smallfile and bigfile tablespaces.

What Happens If I Add a Data File to an Existing Tablespace?

The target database attempts to add a matching data file.

The target database must be able to translate the data file location according to the section above.

2024-08-27T06:51:19.294612+00:00
PDB1(4):Media Recovery Log +RECOC1/SRCDB/partial_archivelog/2024_08_27/thread_2_seq_4.276.1178088679
2024-08-27T06:51:20.268208+00:00
PDB1(4):Successfully added datafile 25 to media recovery
PDB1(4):Datafile #25: '+DATA/TGTCDB_HBZ_FRA/20A568D1FD5DB0A6E0633D01000AC89B/DATAFILE/srctbs01.289.1178088681'

What Happens If I Set a Tablespace Read-Only?

The refreshable clone PDB does not support this. Neither is going the other way: setting a tablespace read-write.

If you do so, the database reports an error:

alter pluggable database pdb2 refresh
*
ERROR at line 1:
ORA-00283: recovery session canceled due to errors
ORA-65339: unsupported operation on the source PDB

From the alert log:

2024-08-28T05:23:02.893946+00:00
PDB2(6):Error! unsupported source PDB operation: 21
2024-08-28T05:23:02.994035+00:00
PDB2(6):Media Recovery failed with error 65339

Operation 21 is setting a tablespace read-only. If you set a tablespace read-write, the database reports operation 20 instead.

PDB2(7):Error! unsupported source PDB operation: 20

You will not be able to refresh the PDB anymore. You must re-create the refreshable clone PDB.

What Happens If I Restart the Source Database?

Refreshable clone PDB does not support restarting the source database.

When you restart the source database, the source database places a special marker in the redo stream. This even happens for a clean shutdown (SHUTDOWN NORMAL). The target CDB does not understand how to recover beyond this marker.

alter pluggable database pdb2 refresh
*
ERROR at line 1:
ORA-00283: recovery session canceled due to errors
ORA-65339: unsupported operation on the source PDB

From the alert log:

2024-08-28T05:27:00.451985+00:00
PDB2(4):Error! unsupported source PDB operation: 3
2024-08-28T05:27:00.710236+00:00
PDB2(4):Media Recovery failed with error 65339

Operation 3 is the source database restart.

You will not be able to refresh the PDB anymore. You must re-create the refreshable clone PDB.

How Do I Drain My Source Database Before Migration?

Right before the migration, when you cut the connection from the source non-CDB to the target PDB, it could be useful to restart the database. But that’s not possible.

I suggest that you:

  • Ensure that the target CDB connects to the source non-CDB using a dedicated service. This applies to the database link that you establish between the two databases.
  • Stop all other services and specify a drain timeout.
  • Shut down the application that connects to the source non-CDB.
  • Kill sessions manually.

Remember that the target database connects to the source database via a database link, so stopping the database listener is not an option. Nor is enabling RESTRICTED SESSION.

Update: Armando managed to perform the migration using restricted session. Check his comment (see below) for details.

What Happens If I Restart the Target Container Database?

You can safely restart the target CDB while you have a refreshable clone PDB. This works fine.

What About NOLOGGING Operations?

You can’t perform NOLOGGING operations on the source database.

Since refreshable clone PDB relies on redo, then a NOLOGGING operation on the source will prevent that data from going to the target. When you try to query the NOLOGGING table on the target database after the migration, you will receive an error:

SQL> select count(*) from t1
       *
ERROR at line 1:
ORA-28304: Oracle encrypted block is corrupt (file # 186, block # 131)
ORA-01110: data file 186:
'+DATA/TGTCDB_HBZ_FRA/20CF181D4A925E06E0633D01000ACB50/DATAFILE/srctbs01.297.117
8266961'
ORA-26040: Data block was loaded using the NOLOGGING option

Thanks to Marcelo for leaving a comment. He suggests that you set the source non-CDB in FORCE LOGGING mode. This is a good idea to avoid this potential nightmare:

alter database force logging;

You can read more about NOLOGGING operations in The Gains and Pains of Nologging Operations (Doc ID 290161.1).

What About Hot Backups?

You can’t perform hot backup operations on the source database.

If you do so, you’ll run into the following error:

2025-11-21T14:31:06.845676+00:00
SALES(4):Error! unsupported source PDB operation: 1
2025-11-21T14:31:07.845923+00:00
SALES(4):Media Recovery failed with error 65339

Please note that I’m not referring to RMAN online backups. I’m talking about the old-school ALTER DATABASE BEGIN BACKUP and ALTER DATABASE END BACKUP commands.

Any restrictions on data types or object types?

No. The refreshable clone is a physical copy of the database, so there are no restrictions on data types or object types.

Services

You must recreate your services after the migration. Neither database managed services nor Clusterware managed services survive the migration.

Further Readin

Summary

Despite these minor restrictions, migration from non-CDB to PDB using refreshable clone PDB and AutoUpgrade is still a very handy method. Knowing the restrictions upfront ensures that you can successfully migrate the database.

Happy migrating!

An Intense Learning Experience with the Product Management Team

Are you considering coming to Oracle DatabaseWorld at CloudWorld, or have you already secured your ticket? Here’s a great opportunity for you.

Oracle CloudWorld

Imagine being in a room with the very people who design and develop the Oracle Database you use every day. The pre-event training at Oracle CloudWorld offers this unique opportunity. My team will be there, and guide you through the upgrade and migration to Oracle Database 23ai.

Optimizing Oracle Database 23ai Multitenant Upgrade

  • On Monday, September 9, 2024
  • From 9:00 to 17:00
  • At Oracle DatabaseWorld at CloudWorld, Las Vegas

Enroll here for the price of $795.

What’s In It?

This is more than just a regular session. It’s an opportunity to dive deep into upgrade and migration to Oracle Database 23ai, understanding the why and the how directly from the source. It is 8 hours of intense training.

By attending our pre-event training, you’ll have the rare chance to learn directly from Mike Dietrich, Rodrigo Jorge, Alex Zaballa, and me. This isn’t just about listening to a keynote – this is hands-on, immersive training where you can ask questions, solve real-world problems, and gain practical skills that you can apply immediately.

This is the agenda that we’ve prepared for you.

Agenda of Optimizing Oracle Database 23ai Multitenant Upgrade

After this training, you have the skills needed to upgrade to Oracle Database 23ai and migrate to the multitenant architecture.

Don’t Miss Out

Register for our pre-event training today!

Supercharge your Oracle CloudWorld experience — start with our pre-event training!

See you there!

The Best Questions on Oracle Multitenant

In late June, my team hosted our 20th webinar in our Virtual Classroom series.

Oracle Webinar

You can now get a curated version of the Q&A. It contains many good questions and answers that give you deeper insight into Oracle Multitenant architecture.

The Best Questions

Consider these questions and answers a little appetizer. You find many more in the Q&A document.

Do I need to convert my database to a PDB before I upgrade to Oracle Database 23ai? No. You can do it as part of the upgrade to Oracle Database 23ai. It is easiest to use AutoUpgrade for the process

Does catcon.pl parallelism require Enterprise Edition like many other parallel options? No. You can use parallel threads in catcon.pl on a Standard Edition 2 database

Any good idea on how to avoid service name collision on a server when two CDBs tries to create a PDB with the same name? We’ve seen several customers implement procedures that create unique names for each PDB. The name of the PDB should not include information about the server or CDB because you often move PDBs around, and then it is impractical to update the PDB name. One customer we worked with created a small PL/SQL function backed by a simple table. When called, the function returned a unique PDB name. The customer used ORDS to expose the function as a web service.

Do you recommend Oracle Fleet Patching and Provisioning (FPP) in a small, but complex, environment? The bigger the environment, the more benefit you get from FPP. Especially if you are using Exadata Database Machine as well. AutoUpgrade is our preferred tool for smaller environments, but the complexity talks towards FPP. FPP comes with a license cost that you must take into consideration as well.

Did You Miss It

In case you missed it, you can find the recording on our YouTube channel and get the slides.

I Love Questions

One of the reasons I like doing webinars is that I get questions from the audience. I love that for several reasons:

  • Many questions about a specific topics is an indication that I didn’t explain a topic very well. I use this as feedback to improve for the next presentation.
  • I can use it to tell where our product has weakness that we can improve.
  • It tells me something about how customers use our product. I always get amazed about the creativity with which our customers use Oracle Database.
  • I find gaps in my own knowledge and I can learn new things.
  • It allows me to help other people.

Please – always ask questions! :)