Oracle Database File System (DBFS) is widely used in Oracle RAC environments for storing shared files such as Oracle Wallets, TDE keystores, ACFS alternatives, GoldenGate files, and application data.

Recently, while troubleshooting an Oracle 19c RAC environment, we encountered a DBFS mount failure that initially appeared to be a Clusterware issue. However, the actual root cause was much simpler—an expired database user password.

This article explains how to identify the problem and restore the DBFS mount.

---

Environment

• Oracle Grid Infrastructure 19c
• Oracle RAC 19c
• DBFS managed by Oracle Clusterware
• Linux

---

Problem

The DBFS resource failed to start on one RAC node.

crsctl start resource dbfs_mount -n pbprdbadm01

Output:

CRS-2672: Attempting to start 'dbfs_mount' on 'pbprdbadm01'
CRS-2674: Start of 'dbfs_mount' on 'pbpr01' failed
CRS-2679: Attempting to clean 'dbfs_mount' on 'pbprdbadm01'
CRS-2681: Clean of 'dbfs_mount' on 'pbpr01' succeeded
CRS-4000: Command Start failed, or completed with errors.

Cluster status showed that DBFS was online on one node but offline on another.

crsctl stat res -t

Example:

--------------------------------------------------------------------------------
Name           Target  State        Server
--------------------------------------------------------------------------------
dbfs_mount
               ONLINE  OFFLINE      pbpr01
               ONLINE  ONLINE       pbpr02

This indicated that the Clusterware resource itself was healthy, but the mount process was failing only on one node.

---

Initial Investigation

The resource configuration looked normal.

crsctl stat res dbfs_mount -p

NAME=dbfs_mount
TYPE=local_resource
ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r--
ACTIONS=
ACTION_SCRIPT=/u01/app/12.1.0.2/grid/crs/script/mount-dbfs.sh
ACTION_TIMEOUT=60
AGENT_FILENAME=%CRS_HOME%/bin/scriptagent
AUTO_START=restore
CHECK_INTERVAL=30
CHECK_TIMEOUT=0
CLEAN_TIMEOUT=60
CSS_CRITICAL=no
DELETE_TIMEOUT=60
DESCRIPTION=
ENABLED=1
IGNORE_TARGET_ON_FAILURE=no
INSTANCE_FAILOVER=1
INTERMEDIATE_TIMEOUT=0
LOAD=1
LOGGING_LEVEL=1
MODIFY_TIMEOUT=60
OFFLINE_CHECK_INTERVAL=0
RESOURCE_GROUP=
RESTART_ATTEMPTS=10
RESTART_DELAY=0
SCRIPT_TIMEOUT=300
SERVER_CATEGORY=
START_CONCURRENCY=0
START_DEPENDENCIES=hard(ora.fsdb.db)pullup(ora.fsdb.db)
START_TIMEOUT=0
STOP_CONCURRENCY=0
STOP_DEPENDENCIES=hard(ora.fsdb.db)
STOP_TIMEOUT=0
TARGET_DEFAULT=default
UPTIME_THRESHOLD=1h
USER_WORKLOAD=no
WORKLOAD_CPU=0
WORKLOAD_CPU_CAP=0
WORKLOAD_MEMORY_MAX=0
WORKLOAD_MEMORY_TARGET=0
 

The action script used by Clusterware was:

ACTION_SCRIPT=/u01/app/12.1.0.2/grid/crs/script/mount-dbfs.sh

The DBFS mount script loads its configuration from:

/etc/oracle/mount-dbfs.conf

This configuration file contains important parameters such as:

• ORACLE_HOME
• DBNAME
• DBFS_USER
• DBFS_PASSWD
• MOUNT_POINT
• WALLET

---

Root Cause

The database account used by DBFS (DBFS_USER) had expired.

Since the DBFS mount script authenticates using this database account, authentication failed and the mount operation returned an error.

Clusterware only reported:

CRS-2674

without indicating the actual database authentication failure.

---

Verification

Connect as SYSDBA and verify the account status.

SELECT username,
       account_status
FROM dba_users
WHERE username='DBFS_USER';

Output:

USERNAME      ACCOUNT_STATUS
------------  ----------------
DBFS_USER     EXPIRED

---

Solution

Step 1 – Reset the password

ALTER USER dbfs_user IDENTIFIED BY Password;

If your environment requires the same password, reset it to the original password.

---

Step 2 – Update the DBFS configuration

If your DBFS environment uses password-based authentication (WALLET=false), you must update the configuration file with the same password that was assigned to the DBFS_USER database account.

The DBFS mount script reads its credentials from:

/etc/oracle/mount-dbfs.conf

Update the following parameters:

DBFS_USER=dbfs_user
DBFS_PASSWD=Password

Important: The value of DBFS_PASSWD must exactly match the current password of the DBFS_USER database account. If the database password is changed or expires and the configuration file is not updated, the DBFS client will fail to authenticate, causing the DBFS mount to fail with generic Clusterware errors such as CRS-2674.

After updating the configuration file, manually test the mount:

/u01/app/12.1.0.2/grid/crs/script/mount-dbfs.sh start

If the filesystem mounts successfully, stop the manual mount and allow Oracle Clusterware to manage the resource:

/u01/app/12.1.0.2/grid/crs/script/mount-dbfs.sh stop

crsctl start resource dbfs_mount -n <node_name>

This ensures that the DBFS filesystem is mounted and monitored correctly by Oracle Clusterware.

---

Step 3 – Test the mount manually

/u01/app/12.1.0.2/grid/crs/script/mount-dbfs.sh start

The filesystem mounted successfully, confirming that the credentials were now valid.

---

Step 4 – Restart the Clusterware resource

If the filesystem was mounted manually, stop it first.

/u01/app/12.1.0.2/grid/crs/script/mount-dbfs.sh stop

Then allow Clusterware to manage the resource.

crsctl start resource dbfs_mount -n pbprdbadm01

Verify:

crsctl stat res -t

Expected output:

dbfs_mount
ONLINE ONLINE pbprdbadm01
ONLINE ONLINE pbprdbadm02

---

Useful Troubleshooting Commands

Check resource status

crsctl stat res dbfs_mount -f

View resource configuration

crsctl stat res dbfs_mount -p

Verify mount

mount | grep dbfs

Check filesystem

df -h | grep dbfs

Check DBFS processes

ps -ef | grep dbfs

---

Why This Happens

Many DBFS deployments still use password-based authentication.

When the password expires:

• Database login fails
• DBFS client cannot authenticate
• Mount operation fails
• Clusterware reports only a generic CRS startup error

Unless you inspect the DBFS configuration or manually execute the mount script, the actual cause can be difficult to identify.

---

Best Practices

• Use an Oracle Wallet instead of storing passwords in configuration files.
• Set an appropriate password policy for service accounts used by DBFS.
• Monitor service account expiration dates.
• Test DBFS mounts after password changes.
• Keep the mount-dbfs.sh script updated according to Oracle Support recommendations.

---

Conclusion

Although the error appeared to be a Clusterware startup issue, the underlying problem was an expired DBFS_USER account. After resetting the password and updating the DBFS configuration file, the DBFS filesystem mounted successfully and the Clusterware resource returned to the ONLINE state.

This serves as a reminder that authentication issues can sometimes present as generic Clusterware failures. Verifying the service account status early in the troubleshooting process can significantly reduce resolution time.