Troubleshooting Oracle Cluster Verification Utility failed

Troubleshooting 11gR2 (11.2.0.1)

With 11gR2 the /etc/host hack to bypass using DNS to resolve the SCAN IP no longer works.

You must have SCAN IP setup in your DNS to install GRID successfully. Some errors

PRVF-4664 : Found inconsistent name resolution entries for SCAN name
and
PRVF-4657 : Name resolution setup check for “rac-scan” (IP address: 192.168.1.202) failed

I was throwing together a quick RAC system on VMware to do some proof of concept testing, when I ran into this pesky error in the last step of the grid install process.

The GUI details said to check the install log: /u01/app/oraInventory/logs/installActions2011-01-16_10-28-20PM.log

Doing so yielded very unhelpful information consisting of:

WARNING:
INFO: Completed Plugin named: Oracle Cluster Verification Utility
INFO: Oracle Cluster Verification Utility failed.

WARNING:
INFO: Completed Plugin named: Oracle Cluster Verification Utility

So I checked failed commands logs: /u01/app/11.2.0/grid/cfgtoollogs/configToolFailedCommands
It only had one command that failed:

/u01/app/11.2.0/grid/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/11.2.0/grid MODE=perform ACTION=configure RERUN=false $*

When I ran that manually, the first error I got was a permissions error:

Unable to complete request:
Fatal error while loading instance XML:
oracle.sysman.emCfg.common.CfwException:
Unable to unmarshal instance xml=/u01/app/11.2.0/grid/inventory/ContentsXML/ConfigXML/oracle.assistants.asm.11_2_0_1_0.CFM.3.inst.xml:
java.io.FileNotFoundException:
/u01/app/11.2.0/grid/inventory/ContentsXML/ConfigXML/oracle.assistants.asm.11_2_0_1_0.CFM.3.inst.xml (Permission denied)

After fixing the permissions on /u01/app/11.2.0/grid/inventory

I ran the script again and got a little further this time but it still failed.
Checking the latest log now yielded PRVF-4657 and PRVF-4664

/u01/app/11.2.0/grid/cfgtoollogs/oui/configActions2011-01-16_11-55-42-PM.log

Checking Single Client Access Name (SCAN)…

Checking name resolution setup for “nil-scan”…

ERROR:
PRVF-4657 : Name resolution setup check for “nil-scan” (IP address: 192.168.1.202) failed

ERROR:
PRVF-4657 : Name resolution setup check for “nil-scan” (IP address: 192.168.1.203) failed

ERROR:
PRVF-4664 : Found inconsistent name resolution entries for SCAN name “nil-scan”

Verification of SCAN VIP and Listener setup failed

At first I thought I must have misconfigured my DNS but after checking the DNS responses with dig and nslookup on both nodes:

dig @192.168.1.101 nil-scan.db
nslookup 192.168.1.201
nslookup 192.168.1.202
nslookup 192.168.1.203

There were no inconsistencies. After checking some other network settings and configurations I still couldn’t find anything that was out of place or not working so I started going over my installation notes and that’s when I found the issue.
Simple and easy to fix but easily overlooked.

My /etc/host file looked like this:

# Public
192.168.1.101 nil1.db nil1
192.168.1.102 nil2.db nil2

# Private
192.168.0.101 nil1-priv.db nil1-priv
192.168.0.102 nil2-priv.db nil2-priv

# Virtual
192.168.1.111 nil1-vip.db nil1-vip
192.168.1.112 nil2-vip.db nil2-vip

# SCAN
192.168.1.201 nil-scan.db nil-scan
192.168.1.202 nil-scan.db nil-scan
192.168.1.203 nil-scan.db nil-scan

I always have the scan ip’s in my /etc/host file for quick reference but have them commented out. I must have gotten distracted while entering them in this time because they weren’t commented out as they should have been:

# Public
192.168.1.101 nil1.db nil1
192.168.1.102 nil2.db nil2

# Private
192.168.0.101 nil1-priv.db nil1-priv
192.168.0.102 nil2-priv.db nil2-priv

# Virtual
192.168.1.111 nil1-vip.db nil1-vip
192.168.1.112 nil2-vip.db nil2-vip

# SCAN
#192.168.1.201 nil-scan.db nil-scan
#192.168.1.202 nil-scan.db nil-scan
#192.168.1.203 nil-scan.db nil-scan

Once I did that I clicked the retry button on the installer and the CVU finished successfully.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s