Adding New node to 11g R2 Cluster

###########################################-----#########################
PART 1. PRE Node add setup and tests
###########################################-----#########################

Install all required rpms and configure new node as existing nodes were done.
update /etc/hosts file on all 3 nodes and add new node hostname and IPs in hosts file.

/etc/hosts file contents

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost

########Public ##############
132.35.21.177 rac1.localdomain rac1
132.35.21.178 rac2.localdomain rac2
132.35.21.179 rac3.localdomain rac3
########Private ##############
10.10.10.1 rac1-priv.localdomain rac1-priv
10.10.10.2 rac2-priv.localdomain rac2-priv
10.10.10.3 rac3-priv.localdomain rac3-priv
########Virtual ##############
132.35.21.187 rac1-vip.localdomain rac1-vip
132.35.21.188 rac2-vip.localdomain rac2-vip
132.35.21.189 rac3-vip.localdomain rac3-vip
########SCAN ##############
132.35.21.198 racscan.localdomain racscan

1. As oracle user use ssh on all three nodes to add rac3 to known host list to ensure passwordless connectivity
On existing nodes

ssh rac3
ssh rac3-priv

On new node (rac3)

ssh rac1
ssh rac2
ssh rac1-priv
ssh rac2-priv

2. [oracle@rac3 ~]$ /media/grid/runcluvfy.sh comp peer -refnode rac1 -n rac3 -orainv oinstall -osdba oinstall -verbose

some tests will fail because new node isn't 100% identical to existing nodes. e.g space etc. You just check those parameters which are required to be equal for cluster
like rpms, network setting etc.
You will definately get this message but you can safely ignore it

###########################################-----#########################
PART 2. Extending Cluster Home new node 
###########################################-----#########################
From any existing node run this command from Grid Home as oracle user

1. [oracle@rac1 bin]$GI_HOME/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={rac3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac3-vip}"

2. When prompt Login as root on new node (rac3) and run root scripts

/u01/app/oraInventory/orainstRoot.sh

/u01/app/11.2.0/grid/root.sh

3. Verify that the new node has been added to the cluster:

$GI_HOME/bin/cluvfy stage -post nodeadd -n -verbose

######################################-------#####################################
PART 3. Extending Oracle Database RAC to new cluster node
#######################################-------#####################################

1. From an existing node from ORACLE_HOME in the cluster as the 'oracle' user:

[oracle@rac1 ]$/u01/app/oracle/product/11.2.0/dbhome_1/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={rac3}"

2. On new node login as root and run root.sh as prompted.

[root@rac3 ~]# /u01/app/oracle/product/11.2.0/dbhome_1/root.sh

3. Update node list in cluster

[oracle@rac1 ]$/u01/app/oracle/product/11.2.0/dbhome_1/oui/bin/runInstaller -silent -updateNodeList ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 "CLUSTER_NODES={rac1,rac2,rac3}"
-----------------------------------------------------------
This step is required if you are using 2 different groups for managing grid infrastructure and database. if you are using same group and user to manage both cluster and db then you don't need
Set ORACLE_HOME and ensure you are using the 'oracle' account user.

Note: Ensure permissions for Oracle executable are 6751, if not, then as root user:
cd $ORACLE_HOME/bin
chgrp asmadmin oracle
chmod 6751 oracle
ls -l oracle
----------------------------------------------------------
4. On any existing node, run DBCA to add the new instance from GUI (RECOMMENDED) which is easiest method

[root@rac1 ~]/u01/app/oracle/product/11.2.0/dbhome_1/bin/dbca
or
from command line run dbca in silent mode

[root@rac1 ~]$/u01/app/oracle/product/11.2.0/dbhome_1/bin/dbca -silent -addInstance -nodeList -gdbName -instanceName -sysDBAUserName sys -sysDBAPassword