Introduction
In a highly available database system like Oracle RAC, node failures are not a matter of if, but when. Hardware issues, network glitches, OS crashes, or storage interruptions can all cause a node to go down.
What makes Oracle RAC powerful is how gracefully it handles these failures.
The secret lies in three critical mechanisms:
Eviction, Failover, and Recovery
These three processes ensure that your database remains available, consistent, and resilient, even when a node suddenly fails.
In this article, we will break down Eviction, Failover, and Recovery in a simple and practical way so you can clearly understand what happens behind the scenes.
Why Failure Handling is Critical in RAC
In a RAC cluster:
- Multiple instances run on different nodes
- All instances access the same shared storage
- Applications connect through services across nodes
If one node crashes without proper coordination, it could lead to:
❌ Data corruption
❌ Split-brain scenario
❌ Lost transactions
❌ Application downtime
To prevent this, Oracle RAC uses Eviction, Failover, and Recovery mechanisms.
Step 1: Eviction – Protecting Cluster Integrity
What is Eviction?
Eviction is the process where a problematic or unresponsive node is forcibly removed from the cluster.
👉 This is done to protect the rest of the cluster from inconsistent behavior.
When does eviction happen?
Eviction can occur due to:
- Network heartbeat failure
- I/O subsystem issues
- CPU starvation
- OS hang
- Voting disk communication failure
Who performs eviction?
The Cluster Synchronization Services (CSSD) process monitors all nodes using heartbeat signals.
If a node stops responding:
👉 CSSD decides to evict the node
What happens during eviction?
When eviction is triggered:
- The node is removed from cluster membership
- The node’s processes are terminated
- Access to shared resources is revoked
- Surviving nodes continue operations
👉 This is sometimes called a “node fencing” mechanism
Why eviction is important
Eviction ensures:
✔ No split-brain scenario
✔ No two nodes updating same data independently
✔ Cluster consistency is preserved
Step 2: Failover – Keeping Services Available
What is Failover?
Once a node is evicted, the next step is Failover.
Failover means:
Moving database services, connections, and workloads from the failed node to surviving nodes.
What fails over in RAC?
After eviction, the following can failover:
- Database services
- Application connections
- Sessions (depending on configuration)
- Jobs and workloads
Types of Failover
1️⃣ Service Failover
Oracle automatically relocates services to other nodes.
Example:
Service SALES_APP was running on node1
node1 fails → SALES_APP moves to node2
2️⃣ Connection Failover
Handled using:
- TAF (Transparent Application Failover)
- FCF (Fast Connection Failover)
- TAC (Transparent Application Continuity)
These features allow applications to:
✔ Reconnect automatically
✔ Resume operations
✔ Continue transactions safely
3️⃣ Workload Failover
With policy-managed RAC, workloads are automatically redistributed using:
- Server pools
- Load balancing
Step 3: Recovery – Bringing Data Back to Consistency
What is Recovery?
After eviction and failover, Oracle must ensure data consistency.
This is where Recovery comes in.
Recovery is the process of:
Rolling forward and rolling back transactions to bring the database to a consistent state.
Types of Recovery in RAC
1️⃣ Instance Recovery
When a node fails:
- Its redo logs are used by surviving instances
- Dirty buffers are applied
- Uncommitted transactions are rolled back
👉 This is called Instance Recovery
2️⃣ Crash Recovery
If the entire instance crashes, Oracle performs:
✔ Roll forward using redo logs
✔ Rollback of incomplete transactions
3️⃣ Cache Recovery
The Global Cache Service (GCS) ensures that:
- Cached blocks from the failed node are recovered
- Ownership is reassigned
Full Failure Flow in RAC
Let’s combine everything into one flow:
Step-by-step failure handling:
- Node stops responding
- CSSD detects heartbeat failure
- Node is evicted from cluster
- Services are failed over to surviving nodes
- Surviving nodes perform instance recovery
- Applications reconnect and continue
👉 This entire process usually completes within seconds
Real-Life Example
Let’s say your production cluster has:
- Node1
- Node2
- Node3
Application is connected to Node1
Suddenly:
- Node1 crashes
What happens?
✔ Node1 is evicted
✔ Service moves to Node2
✔ Sessions reconnect via SCAN
✔ Instance recovery starts
✔ Transactions are recovered
👉 Users may only see a few seconds delay
How to Monitor Eviction, Failover, and Recovery
DBAs can monitor these processes using:
Logs
alert.logocssd.logcrsd.log
Views
GV$INSTANCEGV$SESSIONGV$ACTIVE_SERVICES
Commands
crsctl stat res -t
srvctl status database -d <db_name>
Best Practices for Handling Node Failures
Here are key DBA recommendations:
✔ Use SCAN listeners
Ensures automatic connection failover
✔ Configure services properly
Use preferred and available instances
✔ Enable Application Continuity (TAC)
For seamless transaction replay
✔ Use Fast Connection Failover (FCF)
For quick reconnection
✔ Monitor interconnect network
Prevent false evictions
Easy Way to Remember
Here’s a simple way to remember:
Eviction → Remove bad node
Failover → Move workload
Recovery → Fix data
Conclusion
Oracle RAC is designed for maximum availability, and the combination of:
Eviction, Failover, and Recovery
ensures that your database continues running even during unexpected failures.
These mechanisms guarantee:
✔ Data consistency
✔ Minimal downtime
✔ Seamless user experience
✔ Enterprise-grade reliability
Understanding how Eviction, Failover, and Recovery work will help you:
- Troubleshoot RAC failures faster
- Design resilient systems
- Improve production stability
