I had one of those patching nights again last weekend — the kind where the maintenance window is booked, the pre-checks are green, and you’re one datapatch -verbose away from calling it done. Then it stalls halfway through, and you’re staring at a rollback that refuses to finish.
The task itself was routine on paper: move a 19c database from Release Update 19.27.0.0.0 to 19.31.0.0.0 (the April 2026 RU, patch 39034528), while also rolling back an old interim patch, 37790293, that had been sitting in the registry since the 19.25 days fixing a cross-shard query bug. Two moving pieces, one datapatch run, standard stuff.
The First Sign of Trouble
The binary registry updated cleanly. Datapatch reported the patch installation itself as complete — two patches installed, no complaints there. It’s only when it gets to validating the logfiles that things went sideways:
Patch 37790293 rollback: WITH ERRORS (RETRYABLE)
-> Error at line 1257: script rdbms/admin/catgwmcat.sql
- ORA-04021: timeout occurred while waiting to lock object SYS.DBMS_AQADM_SYS
Patch 39034528 apply: WITH ERRORS (PREV PATCH)
That second line is worth pausing on, because it’s misleading if you don’t know the pattern. The 39034528 apply didn’t actually fail on its own merits — its own logfile showed no errors at all. It was marked WITH ERRORS purely because it was queued behind the failed rollback of 37790293. Datapatch processes patches in a dependency order, and when one step in the queue errors out, downstream steps get flagged too, even if their SQL executed cleanly. I’ve seen this trip people up before: they go digging through the 39034528 logfile looking for a bug that isn’t there.
The real problem was upstream, in catgwmcat.sql, which is part of the Advanced Queuing catalog scripts. An ORA-04021 means exactly what it says — something else in the database held a DDL lock on DBMS_AQADM_SYS long enough that datapatch’s own DDL attempt timed out waiting for it, and by default that wait isn’t long.
Finding the Real Culprit
ORA-04021 errors are almost never about the object itself being broken; they’re about contention. Something else has a lock or a library cache pin on that package while your patch script is trying to touch it. The instinct with these is to go looking for dba_ddl_locks or the library cache pin views, then correlate the SID back to v$session.
The query I actually ran to pull that back was a straightforward join against dba_ddl_locks:
SELECT l.session_id, s.serial#, s.username, s.program, s.machine, s.status
FROM dba_ddl_locks l
JOIN v$session s ON s.sid = l.session_id
WHERE l.name = 'DBMS_AQADM_SYS';
If that comes back empty — which happens when the contention is a library cache pin rather than a DDL lock — the fallback is to go one level deeper:
SELECT kglhdpar, kglnaown, kglnaobj, s.sid, s.serial#, s.username, s.program
FROM x$kglpn p, v$session s
WHERE p.kglpnuse = s.saddr
AND kglnaobj = 'DBMS_AQADM_SYS';
In my case, v$session turned up two things holding activity against AQ-adjacent objects at that moment:
- A
SYSsqlplus session — turned out to be the datapatch session itself, a self-reference, not the actual blocker. - A pair of duplicate-looking rows for SID 7348, user
TCTDBS, running under a scheduler slave process labeled(J005).
That second one was the real story. TCTDBS is an application schema, and J005 is one of the job queue slave processes Oracle spins up to execute DBMS_SCHEDULER or legacy DBMS_JOB jobs. Somewhere in that application’s job scheduling, a job was actively running and had touched AQ dictionary objects — enough to hold a lock that collided directly with what catgwmcat.sql needed during the AQ rollback.
This is a pattern worth remembering: on a live production instance, application job schedulers keep running independently of your patch window unless you explicitly quiesce them. Nobody warns you about this in the patch README. You find out at 11:37 PM.
Killing the Session, Not the Instance
Once I had the SID and serial#, the fix itself was mechanically simple:
ALTER SYSTEM KILL SESSION '7348,38073' IMMEDIATE;
The one thing I was careful about — and something worth flagging for anyone doing this under pressure — is not confusing your own patching session with the blocker. The SYS sqlplus session showing up in the same query result was mine, the one actually driving datapatch. Killing that would have aborted the whole patching run mid-flight, which is a much worse position to be in than a retryable AQ lock. Always confirm which SID belongs to your own terminal before you start issuing kill commands against a result set that has more than one row in it.
With the TCTDBS job session cleared, I re-ran datapatch. Same retry queue, same two patches, but this time:
Patch 37790293 rollback: SUCCESS
Patch 39034528 apply: SUCCESS
Clean finish, both patches confirmed in dba_registry_sqlpatch.
A Note on the Registry History
One thing that catches people off guard afterward: querying dba_registry_sqlpatch still shows the earlier failed attempt as a row in the history, alongside the successful one. You’ll see something like:
39034528 RU APPLY SUCCESS 2026-09-06 00:02:13
37790293 INTERIM ROLLBACK SUCCESS 2026-09-06 00:01:51
39034528 RU APPLY WITH ERRORS (PREV PATCH) 2026-09-05 23:54:03
That’s not a sign of a lingering problem. Datapatch doesn’t overwrite history — it appends. The row that matters is the most recent one by action_time, and as long as that reads SUCCESS, the patch is properly registered. I’ve had people re-open change tickets because they saw the WITH ERRORS row further down the list without checking the timestamp order.
Post-Patch Checklist
Before closing out the maintenance window, I ran through the usual cleanup:
- Checked
dba_objectsfor anything leftINVALIDafter the patch cycle, and ranutlrp.sqlwhere needed. - Restored
job_queue_processesto its original value, since I’d set it to 0 mid-patch to stop new job slaves from spawning while I dealt with the existing one. - Flagged the TCTDBS application team that their job on session 7348 had been killed mid-execution, so they could check for a partial run and re-trigger it if necessary.
That last point matters more than it sounds. Killing a job session clears your lock, but it doesn’t make the job’s own state consistent on the application side. That’s not your problem to fix as the DBA, but it is your responsibility to tell the people who own that schema what happened and when.
The Takeaway
ORA-04021 during catgwmcat.sql in a datapatch run is, in my experience, almost always contention from something outside the patch itself — job schedulers, AQ propagation, or another session with a stale pin on the AQ catalog objects. The fix isn’t a patch bug workaround; it’s identifying and clearing the blocking session, ideally after confirming with the application owners that it’s safe to interrupt. On a production banking system, that confirmation step isn’t optional — it’s the difference between a clean patch window and an uncomfortable conversation the next morning.
If you’re about to run a 19c RU apply with interim patches queued for rollback, it’s worth checking dba_scheduler_running_jobs and v$session for active AQ-touching sessions before you even start, rather than discovering them the way I did.
Critical Patch Updates, Critical Security Patch Updates, Security Alerts and Bulletins
