Introduction
Oracle RAC health check is one of the most important tasks every DBA should perform to ensure cluster stability and high availability. In Oracle RAC and Clusterware environments, regular health checks help detect issues early, maintain performance, and avoid unexpected outages. This guide provides a complete checklist of what to review in Oracle RAC and Clusterware — and how to automate those reviews for consistent reliability.
Why a Health-Check Matters
A health-check is more than just “everything is up”; it’s about proactively finding early warning signs, ensuring configuration alignment, and reducing risk. Oracle themselves provide frameworks such as the Oracle Cluster Health Advisor for monitoring RAC/Clusterware issues.
By having regular reviews you:
- Detect configuration drift or mis‐alignment
- Identify performance or availability risk before it materialises
- Ensure the cluster topology, interconnects, storage and node-health are all in check
- Automate repetitive tasks so your team focuses on real fixes, not firefighting
What to Review: The Checklist
Here’s a structured checklist divided by subsystem. Use this when you wake up each day (or week) to ensure your RAC environment is healthy.
1. Cluster-Node & Operating System
- Check each cluster node is up and responsive (ping, cluster membership)
- Confirm the network interconnect (private interconnect) is stable with no high latency or dropped packets
- Validate OS resource usage: CPU, memory, IO, swap, load averages
- Check OS patch level and kernel parameters align with Oracle docs
- Review node crash or panic logs for unusual events
2. Oracle Clusterware & Grid Infrastructure
- Ensure all clusterware services are running on each node
- Validate VIPs, Scan IPs, GSD, GES processes are healthy
- Check voting disk and OCR status: no corruption, correct access permissions
- Use tools like
crsctlandolsnodesto verify cluster node status - For Solaris/Veritas users, review advanced health checks such as LLT, I/O fencing, PrivNIC and I/O fencing logs. veritas.com
- Use the Cluster Health Monitor (CHM) / OCLUMON to review collected metrics and node health. Oracle Docs
3. Oracle RAC Database Instances
- Check that each RAC instance is open, services registered, and performance baselines are within expected thresholds
- Review alert logs for recurring ORA warnings or errors
- Validate listener status, service registrations and correct load-balancing distributions
- Monitor interconnect waits and global cache coherency waits (e.g., gc buffer busy, global cache cr blocks)
- Review ASM disk group status:
ASMCAorV$ASM_*views for free space, redundancy, rebalance operations - Check that archive/log switch rates are healthy and redo logs are not stressed
4. Shared Storage & Interconnects
- Confirm all ASM disks or shared storage LUNs are visible on all nodes, with identical paths and no missing/mismatched devices
- Check I/O latency for shared storage: if LUN latency creeps up, RAC performance and cache fusion may suffer
- Validate interconnect reliability: packet drops, network timeouts, NIC failures can impact cache fusion
- Verify fencing mechanisms are functional (node eviction, split-brain prevention)
5. Configuration & Best Practices
- Confirm current cluster and database versions are supported and patched — mismatch can lead to instability
- Check clusterware version is ≥ RAC database version. For upgrades, Oracle mandates the clusterware be same or newer. Oracle Docs
- Review parameter consistency across RAC nodes (for example SGA, PGA, initialization parameters)
- Confirm time-synchronisation across nodes (NTP) — skew can cause cluster membership issues
- Review backups and cluster-aware backup solutions, verify restore‐test has been done
- Validate DR/Standby configurations and roles if applicable
How to Automate the Health-Check
Manual checks are fine, but automation means you catch issues early, generate reports and reduce human error. Here are steps to automate:
Use Oracle’s Built-in Tools
- Oracle’s Automatic Health Framework (AHF) and Health Monitor: you can run checks via
DBMS_HMor via Enterprise Manager. - Cluster Health Advisor (mentioned above) runs continuously and flags anomalies.
Use ORAchk / EXAchk / CVU for RAC
- For RAC environments you can schedule regular runs of tools like
orachk/exachk(depending on platform) in on-demand mode. - Configure via cron/Windows Task Scheduler to run nightly, digest outputs, and alert on warnings or failures
Custom Scripting for RAC/Clusterware
- Write shell or PowerShell scripts to check key outputs like
crsctl check cluster
crsctl stat res -t
asmcmd lsdg
oclumon dumpnodeview
- Parse logs (alert logs, clusterware logs) for keywords (ORA-, CRS-, GES-, voting disk errors) and send summary emails
- Use SNMP traps or monitoring platforms (e.g., Grafana, Prometheus) to capture metrics and alert on thresholds
Reporting & Dashboards
- Generate HTML or PDF reports nightly/weekly with a summary of node status, alert count, disk group space, I/O latency, cluster events
- Use dashboards to trend metrics over time: node load, interconnect latency, redo generation, cache fusion metrics
- Have an SLA-based view: e.g., “no critical ORA errors in the last 7 days”, “interconnect latency > 1ms for <5% of time”, “ASM free space > 20%”
Best Practices & Tips
- Always test your automation scripts in a non-production environment first
- Alert thresholds should be realistic — fine-tune based on your cluster’s baseline
- Keep your clusterware and RAC versions supported and patched
- Make sure your private interconnect is isolated and redundant
- Regularly review log files as automation may only catch known patterns — human review still matters
- Document your health-check automation process and keep scripts in version control
Summary
Maintaining a healthy Oracle RAC + Clusterware environment is critical for high availability and performance. With the checklist above you can systematically review the major areas: cluster nodes, storage/interconnects, database instances, and configuration. By automating these checks you’ll free your team for higher-value tasks, and raise the barrier for failure.
Start with manual reviews, build trust in the process, then automate everything you can. Your future self (and your on-call nights) will thank you.
