Adding New Node to Oracle 11gR2 RAC

Phase I – Extending Oracle Clusterware to a new cluster node
1. Make physical connections, install the OS. Follow article "11GR2 GRID INFRASTRUCTURE INSTALLATION FAILS WHEN RUNNING ROOT.SH ON NODE 2 OF RAC [ID 1059847.1]" to ensure successful completion of root.sh!!

2. Create Oracle accounts, and setup SSH among the new node and the existing cluster nodes. Follow article "How To Configure SSH for a RAC Installation [ID 300548.1]" for correct procedure for SSH setup! EACH NODE MUST BE VISITED TO ENSURE THEY ARE ADDED TO KNOWN_HOSTS FILE!!

3. Verify the requirements for cluster node addition using the Cluster Verification Utility (CVU). From an existing cluster node:

$> $GI_HOME/bin/cluvfy stage -post hwos -n -verbose

4. Compare an existing node with the new node(s) to be added:

$> $GI_HOME/bin/cluvfy comp peer -refnode -n -orainv oinstall -osdba dba -verbose

5. Verify the integrity of the cluster and new node by running from an existing cluster node:

$GI_HOME/bin/cluvfy stage -pre nodeadd -n -fixup -verbose

6. Add the new node by running the following from an existing cluster node:

a. Not using GNS

$GI_HOME/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={}"

b. Using GNS

$GI_HOME/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={}"

Run the root scripts when prompted.

7. Verify that the new node has been added to the cluster:

$GI_HOME/bin/cluvfy stage -post nodeadd -n -verbose

Phase II – Extending Oracle Database RAC to new cluster node
8a. Using Clone Process

i. Use 'tar' to archive an existing DB home, and extract to the same location on the new node

ii. On the new node run:

perl $ORACLE_HOME/clone/bin/clone.pl '-O"CLUSTER_NODES={,}"' '-O"LOCAL_NODE="' ORACLE_BASE=$ORACLE_BASE ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=OraDb11g_home1 '-O-noConfig'

iii. On the existing node where the DB home was cloned, run:

$ORACLE_HOME/oui/bin/runInstaller –updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={,}"

OR

8b.Using addNode.sh process (RECOMMENDED)

i. From an existing node in the cluster as the 'oracle' user:

$> $ORACLE_HOME/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={}"

9. On the new node run the root.sh as prompted.

10. Set ORACLE_HOME and ensure you are using the 'oracle' account user.

Note: Ensure permissions for Oracle executable are 6751, if not, then as root user:
cd $ORACLE_HOME/bin
chgrp asmadmin oracle
chmod 6751 oracle
ls -l oracle

11. On any existing node, run DBCA ($ORACLE_HOME/bin/dbca) to add the new instance:

$ORACLE_HOME/bin/dbca -silent -addInstance -nodeList -gdbName -instanceName -sysDBAUserName sys -sysDBAPassword

NOTES: A. Ensure command is run from existing node with same or less memory, otherwise command will fail due to insufficient memory to support database memory structures. Also ensure that the log file is checked for actual success since it can differ from what is displayed at the screen.
B. Anytime when a patch is applied to the database ORACLE_HOME, please ensure above ownership and permission is corrected after the patch.

12. Verify the administrator privileges on the new node by running on existing node:

$ORACLE_HOME/bin/cluvfy comp admprv -o db_config -d $ORACLE_HOME -n -verbose

13. For an Admin-Managed Cluster, add the new instance to services, or create additional services. For a Policy-Managed Cluster, verify the instance has been added to an existing server pool.

14. Setup OCM in Cloned Homes

a. Delete all subdirectories to remove previously configured host:

$> rm -rf $ORACLE_HOME/ccr/hosts/*

b. Move (no copy) both 'grid' and 'oracle' homes:

$> mv $ORACLE_HOME/ccr/inventory/core.jar $ORACLE_HOME/ccr/inventory/pending/core.jar

c. Configure OCM for the cloned home on the new node:

$> $ORACLE_HOME/ccr/bin/configCCR -a