For anyone who has the misfortune of having to install a multi-node RAC cluster of Oracle 11g R2 on Solaris 10 (in my case, SPARC 64-bit) with an EMC PowerPath, hopefully these notes I’ve collected will come in useful. I certainly suffered plenty trying to get this install to work and web searches didn’t turn up very many useful results at the time.

Make sure you set tunable limits correctly.

Previously, you would edit /etc/system to change tunable parameters then reboot the system, but starting in Solaris 10, there’s a new “projects” system where tunables can be configured. For my installation, the cluster nodes had 16GB of RAM each, so I used these settings (as root):

$ projmod -U oracle -sK "process.max-file-descriptor=(priv,4096,deny)" user.oracle
$ projmod -U oracle -sK "project.max-msg-ids=(priv,256,deny)" user.oracle
$ projmod -U oracle -sK "project.max-sem-ids=(priv,256,deny)" user.oracle
$ projmod -U oracle -sK "project.max-shm-ids=(priv,256,deny)" user.oracle
$ projmod -U oracle -sK "project.max-shm-memory=(priv,16GB,deny)" user.oracle

You can check these settings by inspecting /etc/project and looking for the user.oracle entry.

Also, when running in a RAC configuration, some Oracle processes like crsd run as root, so these will be necessary as well:

$ projmod -sK "project.max-shm-ids=(priv,256,deny)" system
$ projmod -sK "project.max-shm-memory=(priv,16GB,deny)" system

This appears to be a catch-all way of ensuring these settings are in use for all users:

$ projmod -sK "project.max-shm-ids=(priv,256,deny)" default
$ projmod -sK "project.max-shm-memory=(priv,16GB,deny)" default

Failing to set these settings for system and default seems to result in Oracle “ORA-01102: cannot mount database in EXCLUSIVE mode” errors when trying to start a database instance.

NB: There may be circumstances when you still need to configure these tunables in /etc/system. For more information, see Oracle Metalink 435464.1 ENABLING SOLARIS PROJECT SETTINGS FOR CRS.

Failure to create the DATA diskgroup using ASM

I wasted a good 6 hours trying to understand why I kept getting the following error during the 11g R2 grid package install:

ORA-15020: discovered duplicate ASM disk "DATA_0000"

Turns out, I had only set the permissions on /dev/rdsk/emcpower* correctly on one of the nodes in the cluster. The permissions need to be correct on all nodes in the cluster, as the installer operates on all cluster nodes as part of the install – duh! Beware of this gotcha, it can be very frustrating.

WARNING: oradism did not start up correctly.

In diag/rdbms/*/$ORACLE_SID/trace, in the alert_${ORACLE_SID}.log, you may find this message:

WARNING: oradism did not start up correctly.
  Dynamic ISM can not be locked.
----------------------------------------
oradism creation failed for unknown reasons 0 8 105

This is apparently a bug described in Oracle Metalink 374367.1. By default, the Oracle installer installs the file with ownership oracle:oinstall and mode 0750. It needs to be suid root, sgid dba. The solution is to perform the following steps (as root):

$ cd $ORACLE_HOME/bin
$ chown root:dba oradism
$ chmod 6550 oradism

Then, restart the database.

Back up the Enterprise Manager encryption key!

The Enterprise Manager runs on port 1158, using HTTPS. The encryption key is located here:

$ORACLE_HOME/*/sysman/config/emkey.ora

Back this key up, because without it, the Enterprise Manager data can’t be accessed.

ERROR: NMO not setuid-root (Unix-only)

When using Enterprise Manager, you might get the following error message:

ERROR: NMO not setuid-root (Unix-only)

This occurs because the NMO binaries need to be setuid-root, as the error explains. Use these steps to correct the problem:

$ cd $ORACLE_HOME/bin
$ chown root nmb nmhs nmo
$ chmod 6750 nmb nmhs nmo

---

Have you installed a multi-node RAC cluster using Oracle 11g R2 on Solaris 10? Did you discover any gotchas that prevented a successful installation “out of the box”? Please, share them in the comments below so that others can benefit from our pain!