When cluvfy Says Your GI Home Is Broken: A Real PRVG-2031 / PRVG-11960 Fix

I sat down last week to run a routine pre-patch check on a two-node RAC cluster before applying the latest Grid Infrastructure patch, and node2 threw back something I hadn’t seen quite this messy before. Over a thousand lines of PRVG-2031 errors, one after another, every single binary under $GRID_HOME/bin and $GRID_HOME/lib flagged as owned by grid when cluvfy expected root. Somewhere buried in that wall of text sat a second, quieter error code — PRVG-11960 — pointing at one specific file: oradism.

If you’ve patched enough RAC clusters, you’ve probably run into some flavor of this. But the way it played out here, and the wrong turns I took chasing it, felt worth writing down.

The setup

Standard stuff. GI 23.26.0.0, two-node cluster, about to patch. First step, always, is cluvfy stage -pre patch as the grid user before touching anything with opatch. Node1 sailed through clean. Node2 came back like this:

Software home: /u01/app/23.26.0.0/grid ...FAILED (PRVG-2031, PRVG-11960)

Followed by roughly a thousand lines, one per binary, each saying the same thing: expected owner root(0), found grid(54331).

The instinct, and why it’s half right

My first read on this was straightforward: the GI home was sitting in an “unlocked” state. Oracle deliberately changes ownership of the privileged files under bin and lib from root to grid whenever the home gets unlocked — usually via rootcrs.sh -unlock, which happens as part of a normal opatch cycle so the patching tools can write to those files as the grid OS user without needing root for every step. If a previous patch session gets interrupted before the matching -lock step runs, or someone runs -unlock by hand and walks away, you’re left with exactly this signature: hundreds of grid-owned files where root should be.

So the fix, in theory, is one command:

./rootcrs.sh -lock

In practice it took three attempts before it actually did anything. First run: -unlock instead of -lock, by mistake, which just confirmed the broken state rather than fixing it. Second run: -lock completed in about four seconds with no visible progress messages, no CLSRSC-347 success banner, nothing. On top of that, cluvfy’s next run came back with a completely different set of failures — OLR integrity gone, ASM status unverifiable, crsctl unable to read the OCR device, “Permission denied.” For a minute it looked like the stack itself had come apart mid-transition.

That turned out to be a red herring. crsctl check crs showed OHASD, CRSD, CSSD, and EVMD all online the whole time. The mid-lock cluvfy failure was catching the cluster in an inconsistent snapshot during the ownership transition, not an actual outage.

Where it actually went sideways

The real problem was that I couldn’t tell, from the log alone, whether -lock had finished or silently bailed. The log was full of lines like:

Got permissions of file /u01/app/23.26.0.0/grid/bin/ohasd: 0740 owner: root group: oinstall

which look like confirmations but aren’t — that’s the script recording the original state of each file before touching it, not confirming a successful chown afterward. Compare that to a different section of the same log, patching usm/install, which explicitly logged Successful lock ownership on ... after each step. The bin/lib section never got that confirmation line in the log I captured, which sent me down a path of assuming the run had aborted partway.

Here’s the part worth remembering: oradism on the healthy node — node1, the one that was passing cluvfy the whole time — was also grid-owned, with no setuid bit. That immediately broke my working theory that “root + setuid” was some universal requirement I could check node2 against in isolation. It sent me chasing capabilities (getcap) next, on the theory that modern GI might grant oradism its privilege through Linux capabilities instead of the old setuid model. Also empty, on both nodes. Dead end.

The lesson: don’t trust your own model of “what correct looks like” from documentation or memory when you have a second, currently-passing node sitting right there. Diff against it instead of reasoning in the abstract. Once I ran the same find command on node1 and compared file lists side by side, it became obvious that node2’s remaining non-root files (asmcmd, tfactl, rhpctl, onsctl, schema_operations.sh, and a handful of others) were a strict subset of node1’s much longer non-root list. Those are files Oracle’s own crsconfig_fileperms.excl intentionally leaves grid-owned — user-facing CLI utilities, not privileged daemons. Node2’s bin directory was, if anything, in better shape than node1’s by that point.

What actually fixed it

The -lock run had, in fact, worked. The missing success banner was cosmetic — a logging quirk, not a failed operation. oradism came back as:

-r-sr-x---. 1 root oinstall 1138016 Jan 30  2026 oradism

Root-owned, setuid bit set, exactly what PRVG-11960 wanted. Re-running the pre-check confirmed it:

Software home: /u01/app/23.26.0.0/grid ...PASSED
Immutable file attribute check ...PASSED
Pre-check for Patch Application was successful.

Takeaways

A few things I’m carrying forward from this one:

• If cluvfy fails a “Software home” check with a mountain of PRVG-2031 errors right before a patch, check for an unlocked GI home before assuming corruption. rootcrs.sh -lock is the fix, but give it time to actually run — a session that returns to the prompt in under five seconds without progress messages is suspicious, not necessarily broken.
• Don’t assume every file in $GRID_HOME/bin needs to be root-owned. Oracle maintains a deliberate exclusion list for user-facing CLI tools. When in doubt, diff against a node that’s currently passing rather than trusting a mental model of what “correct” should look like.
• A mid-transition cluvfy run can throw alarming, unrelated-looking errors (OCR permission denied, ASM status unknown) that have nothing to do with the actual root cause. Check crsctl check crs directly before chasing those down a separate rabbit hole.
• Ownership and setuid bits don’t tell the whole privilege story on modern Linux — capabilities can substitute for both. Worth ruling out explicitly with getcap rather than assuming from ls -l alone, even though in this case it turned out to be a non-factor.

Patch went through clean after that. Sometimes the fix really is the one-liner you reached for in the first minute — the trick is trusting the evidence over the assumption once things stop looking clean on the way there.