Installing Oracle 11g R2 RAC on Solaris 10

Host OS:                                       Windows 7 professional
Guest OS:                                     Sun Solaris 10 update 10 x86 64bit
Virtualization Tool :               Oracle Virtual Box
Oracle Database:                      Oracle Database 11g R2
 =========================================================
Operating System Installation

Please click on this Link 
Solaris 10 installation on Virtual Box


OS Preparation

Check Package availability

bash-3.2# pkginfo -i SUNWarc SUNWbtool SUNWcsl SUNWhea SUNWlibC SUNWlibm SUNWlibms SUNWsprot SUNWtoo SUNWi1of SUNWi1cs SUNWi15cs

SUNWi1cs SUNWi15cs SUNWxwfnt
system      SUNWarc   Lint Libraries (usr)
system      SUNWbtool CCS tools bundled with SunOS
system      SUNWcsl   Core Solaris, (Shared Libs)
system      SUNWhea   SunOS Header Files
system      SUNWi1of  ISO-8859-1 (Latin-1) Optional Fonts
system      SUNWlibC  Sun Workshop Compilers Bundled libC
system      SUNWlibm  Math & Microtasking Library Headers & Lint Files (Usr)
system      SUNWlibms Math & Microtasking Libraries (Usr)
system      SUNWsprot Solaris Bundled tools
system      SUNWtoo   Programming Tools
system      SUNWxwfnt X Window System platform required fonts
ERROR: information for "SUNWi1cs" was not found
ERROR: information for "SUNWi15cs" was not found

================================================================
Install Missing Packages

bash-3.2# pkgadd -d /cdrom/sol_10_811_x86/Solaris_10/Product SUNWi1cs SUNWi15cs

===============================================================
Edit /etc/hosts file

bash-3.2# gedit /etc/hosts

# Internet host table
#
::1 localhost
127.0.0.1 localhost
#Public
192.168.1.1 racnode1 racnode1.localdomain loghost
192.168.1.2 racnode2 racnode2.localdomain loghost
#
#
#
#Private
192.168.2.1 racnode1-priv racnode1-priv.localdomain
192.168.2.2 racnode2-priv racnode2-priv.localdomain
#
#
#
#Virtual
192.168.1.11 racnode1-vip racnode1-vip.localdomain
192.168.1.12 racnode2-vip racnode1-vip.localdomain
#
#
#
#Scan
192.168.1.50 rac-scan rac-scan.localdomain

==============================================================
Add required groups (minimum 2 oinstall and dba )

groupadd oinstall
groupadd dba
groupadd asmadmin
groupadd asmdba
groupadd asmoper

===============================================================
Add grid and Oracle User
(To keep things simple in this exercise i am just using oracle user and one group oinstall as ASM and Database administrator) if you to follow this blog you only need one user oracle but if you want to use grid for asm admin then you need to create grid user.

bash-3.2# useradd -g oinstall -G asmadmin,asmdba,asmoper -d /export/home/grid -s /bin/bash grid
bash-3.2# useradd -g oinstall -G asmdba,dba -d /export/home/oracle -s /bin/bash oracle

bash-3.2# passwd oracle
New Password:
Re-enter new Password:
passwd: password successfully changed for oracle
bash-3.2# passwd grid
New Password:
Re-enter new Password:

passwd: password successfully changed for grid

=================================================================
Create home directories for Oracle and Grid User

bash-3.2# mkdir /export/home/oracle
bash-3.2# mkdir /export/home/grid
bash-3.2# chown -R oracle:oinstall /export/home/oracle
bash-3.2# chown -R grid:oinstall /export/home/grid

=================================================================

Check users setting

Open a new prompt window and test login with both users grid and oracle
bash-3.2# su - oracle
Oracle Corporation      SunOS 5.10      Generic Patch   January 2005
bash-3.2# exit
logout
bash-3.2# su - grid
Oracle Corporation      SunOS 5.10      Generic Patch   January 2005

================================================================
Disable autofs

bash-3.2# svcadm disable autofs

 ================================================================
Create Directories for installation

• Inventory

bash-3.2# mkdir -p /u01/app/oraInventory
bash-3.2# chown -R oracle:oinstall /u01/app/oraInventory
bash-3.2# chmod -R 775 /u01/app/oraInventory

• Grid

bash-3.2# mkdir -p /u01/11.2.0/grid
bash-3.2# chown -R oracle:oinstall /u01/11.2.0/grid
bash-3.2# chmod -R 775 /u01/11.2.0/grid

• Database

bash-3.2# mkdir -p /u01/app/oracle
bash-3.2# mkdir /u01/app/oracle/cfgtoollogs 
bash-3.2# chown -R oracle:oinstall /u01/app/oracle
bash-3.2# chown -R oracle:oinstall /u01/app/oracle
bash-3.2# mkdir -p /u01/app/oracle/product/11.2.0/db_1
bash-3.2# chown -R oracle:oinstall /u01/app/oracle/product/11.2.0/db_1
bash-3.2# chmod -R 775 /u01/app/oracle/product/11.2.0/db_1

================================================================
Disable Firewall

bash-3.2# svcadm disable ipfilter

 ==========================================================
Add project for Oracle User

bash-3.2# projadd -U oracle oracle

append following line in /etc/user_attr file

oracle::::project=oracle

Add / Update project and processes setting for Oracle User using these commands

bash-3.2# projmod -s -K "project.max-sem-ids=(priv,100,deny)" oracle
bash-3.2# projmod -s -K "process.max-sem-nsems=(priv,256,deny)" oracle
bash-3.2# projmod -s -K "project.max-shm-memory=(priv,4g,deny)" oracle
bash-3.2# projmod -s -K "project.max-shm-ids=(priv,100,deny)" oracle
bash-3.2# projmod -s -K "process.max-file-descriptor=(basic,1024,deny)" oracle

=================================================================
update these values in /etc/system by running these commands in terminal

bash-3.2# echo "set max_nprocs = 30000"  >> /etc/system
bash-3.2# echo "set maxuprc = 16384"     >> /etc/system
bash-3.2# echo "set noexec_user_stack=1"  >> /etc/system
bash-3.2# echo "set semsys:seminfo_semmni=100"  >> /etc/system
bash-3.2# echo "set semsys:seminfo_semmns=1024"  >> /etc/system
bash-3.2# echo "set semsys:seminfo_semmsl=256"  >> /etc/system
bash-3.2# echo "set semsys:seminfo_semvmx=32767"  >> /etc/system
bash-3.2# echo "set shmsys:shminfo_shmmax=4294967296"  >> /etc/system
bash-3.2# echo "set shmsys:shminfo_shmmni =100"  >> /etc/system

==============================================================
TCP and UDP parameters setting

Run these commands to change tcp and udp parameters

bash-3.2# /usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 9000

bash-3.2# /usr/sbin/ndd -set /dev/tcp tcp_largest_anon_port 65500

bash-3.2# /usr/sbin/ndd -set /dev/udp udp_smallest_anon_port 9000

bash-3.2# /usr/sbin/ndd -set /dev/udp udp_largest_anon_port 65500

==========================================================

Add these lines in /etc/inittab the following lines for the TCP and UDP parameters to persist across reboot.

tm::sysinit:/usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 9000 > /dev/console
tm::sysinit:/usr/sbin/ndd -set /dev/tcp tcp_largest_anon_port 65500 > /dev/console
tm::sysinit:/usr/sbin/ndd -set /dev/udp udp_smallest_anon_port 9000 > /dev/console
tm::sysinit:/usr/sbin/ndd -set /dev/udp udp_largest_anon_port 65500 > /dev/console

==============================================================
Automatic SSH configuration

Oracle recommends using OUI to setup ssh user equivalence where users can connect across the node of the cluster without a password. To avoid errors while attaching $OH when the remote node closes a connection prematurely make following changes in /etc/ssh/sshd_config file 

LoginGraceTime 0

After changing the file restart ssh service.

bash-3.2# svcadm restart ssh

===============================================================
Enable Core Dumps

Make sure that Core Dumps are enabled. To check if core dumps are enabled use coreadm command as follows.

bash-3.2# coreadm

global core file pattern:

global core file content: default

init core file pattern: core

init core file content: default

global core dumps: disabled

per-process core dumps: enabled

global setid core dumps: disabled

per-process setid core dumps: disabled

global core dump logging: disabled

bash-3.2#

As root user make the following directory. And enable core dumps as show below.

bash-3.2# mkdir -p /var/cores

coreadm -g /var/cores/%f.%n.%p.%t.core -e global -e global-setid -e log -d process -d proc-setid

===============================================================

Network Time protocol Settings

There are two options for time synchronization: 

• An operating system configured network time protocol (NTP)

or

• Oracle Cluster Time Synchronization Service. 

Oracle Cluster Time Synchronization Service is designed for organizations whose cluster servers are unable to access NTP services. If you use NTP, then the Oracle Cluster Time Synchronization daemon (ctssd) starts up in observer mode. If you do not have NTP daemons, then ctssd starts up in active mode and synchronizes time among cluster members without contacting an external time server. So there are two options

Disable NTP and rely entirely on CTSS:


===============================================================

I am not using NTP so i disabled it

As root execute the command below:

bash-3.2# svcadm disable ntp



================================================================
This Step is Optional

If you want to use ntp and can access a ntp server then follow these steps

Add these parameter in file /etc/inet/ntp.conf 
“slewalways yes” 
“disable pll” 

Restart xntpd using the command

bash-3.2# svcadm restart ntp
===============================================================
bash-3.2# reboot
======================================================================
Check project settings for user oracle

bash-3.2# prctl -n project.max-shm-memory -i project oracle
project: 100: oracle
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
        privileged      4.00GB      -   deny                                 -
        system          16.0EB    max   deny                                 -
bash-3.2#


NOTE: Open terminal and login with oracle user (Atlease one session for oracle user should be opened ) otherwise you will get this error

prctl: oracle: No controllable process found in task, project, or zone.

 ===========================================================
Configure Network Interfaces

Query all interface information

bash-3.2# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
        inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
        ether 0:c:29:21:12:f5

Plumb all interfaces

bash-3.2# ifconfig -a plumb

ifconfig: SIOCSLIFNAME for ip: e1000g0: already exists
bash-3.2#

Query again and you will see newly added network interface

bash-3.2# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
        inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
        ether 0:c:29:21:12:f5
e1000g1: flags=1000842 mtu 1500 index 3
        inet 0.0.0.0 netmask 0
        ether 0:c:29:21:12:ff
bash-3.2#

Configure new interface

bash-3.2# ifconfig e1000g1 192.168.2.1 netmask 255.255.255.0 up
bash-3.2# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
        inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
        ether 0:c:29:21:12:f5
e1000g1: flags=1000843 mtu 1500 index 3
        inet 192.168.2.1 netmask ffffff00 broadcast 10.10.10.255
        ether 0:c:29:21:12:ff
==========================================================
Make changes permanent

bash-3.2# ls -ltr /etc/hostname*
-rw-r--r--   1 root     root           9 Jan 31 06:12 /etc/hostname.e1000g0

Create file for new interface e1000g1

bash-3.2# gedit /etc/hostname.e1000g1

Add hostname in newly created file hostname.e1000g1 for new private interface

racnode1-priv

bash-3.2# ls -ltr /etc/hostname*
-rw-r--r--   1 root     root           9 Jan 31 06:12 /etc/hostname.e1000g0
-rw-r--r--   1 root     root          14 Jan 31 06:46 /etc/hostname.e1000g1

Reboot machine

bash-3.2# reboot
=================================================================
Add and Configure ASM Shared Disks For VMBox

Shutdown virtual machine and create folder for ASM storage (D:\VMBOX\ASM in my case). Open windows command prompt and navigate to ASM folder and issue following commands. 



D:\VMBOX\ASM>VBoxManage createhd --filename asmdisk1.vdi --size 5120 --format VDI --variant Fixed

D:\VMBOX\ASM>VBoxManage createhd --filename asmdisk2.vdi --size 5120 --format VDI --variant Fixed
D:\VMBOX\ASM>VBoxManage createhd --filename asmdisk3.vdi --size 5120 --format VDI --variant Fixed
D:\VMBOX\ASM>VBoxManage createhd --filename asmdisk4.vdi --size 5120 --format VDI --variant Fixed

================================================================
Attach Disks to SCSI storage controller

D:\VMBOX\ASM>VBoxManage storageattach RACNODE1 --storagectl "SCSI" --port 1 --device 0 --type hdd --medium asmdisk1.vdi --mtype shareable

D:\VMBOX\ASM>VBoxManage storageattach RACNODE1 --storagectl "SCSI" --port 2 --device 0 --type hdd --medium asmdisk2.vdi --mtype shareable

D:\VMBOX\ASM>VBoxManage storageattach RACNODE1 --storagectl "SCSI" --port 3 --device 0 --type hdd --medium asmdisk3.vdi --mtype shareable

D:\VMBOX\ASM>VBoxManage storageattach RACNODE1 --storagectl "SCSI" --port 4 --device 0 --type hdd --medium asmdisk4.vdi --mtype shareable

Here RACNODE1 is Virtual Machine Name.

===============================================================
Make Disks Shareable

D:\VMBOX\ASM>VBoxManage modifyhd asmdisk1.vdi --type shareable

D:\VMBOX\ASM>VBoxManage modifyhd asmdisk2.vdi --type shareable

D:\VMBOX\ASM>VBoxManage modifyhd asmdisk3.vdi --type shareable

D:\VMBOX\ASM>VBoxManage modifyhd asmdisk4.vdi --type shareable

=================================================================

Change ownership of hard disks

bash-3.2# chown oracle:oinstall /dev/rdsk/c1t1d0s0
bash-3.2# chown oracle:oinstall /dev/rdsk/c1t2d0s0
bash-3.2# chown oracle:oinstall /dev/rdsk/c1t3d0s0
bash-3.2# chown oracle:oinstall /dev/rdsk/c1t4d0s0

bash-3.2# chmod 660 /dev/rdsk/c1t1d0s0
bash-3.2# chmod 660 /dev/rdsk/c1t2d0s0
bash-3.2# chmod 660 /dev/rdsk/c1t3d0s0
bash-3.2# chmod 660 /dev/rdsk/c1t4d0s0

=================================================================
Format Hard Disks

#format

AVAILABLE DISK SELECTIONS:
       0. c1t0d0
          /pci@0,0/pci15ad,1976@10/sd@0,0
       1. c1t1d0
          /pci@0,0/pci1000,8000@14/sd@1,0
       2. c1t2d0
          /pci@0,0/pci1000,8000@14/sd@2,0
       3. c1t3d0
          /pci@0,0/pci1000,8000@14/sd@3,0
       4. c1t4d0

Specify disk (enter its number): 1
selecting c2t2d0
[disk formatted]
format> p
WARNING - This disk may be in use by an application that has
          modified the fdisk table. Ensure that this disk is
          not currently in use before proceeding to use fdisk.
format> fdisk
No fdisk table exists. The default partition for the disk is:

  a 100% "SOLARIS System" partition

Type "y" to accept the default partition,  otherwise type "n" to edit the
 partition table.
y
format> p


PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
partition> p
Current partition table (original):
Total disk cylinders available: 2607 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders        Size            Blocks
  0 unassigned    wm       0               0         (0/0/0)           0
  1 unassigned    wm       0               0         (0/0/0)           0
  2     backup    wu       0 - 2606       19.97GB    (2607/0/0) 41881455
  3 unassigned    wm       0               0         (0/0/0)           0
  4 unassigned    wm       0               0         (0/0/0)           0
  5 unassigned    wm       0               0         (0/0/0)           0
  6 unassigned    wm       0               0         (0/0/0)           0
  7 unassigned    wm       0               0         (0/0/0)           0
  8       boot    wu       0 -    0        7.84MB    (1/0/0)       16065
  9 unassigned    wm       0               0         (0/0/0)           0

partition> 0
Part      Tag    Flag     Cylinders        Size            Blocks
  0 unassigned    wm       0               0         (0/0/0)           0

Enter partition id tag[unassigned]:
Enter partition permission flags[wm]:
Enter new starting cyl[0]: 1
Enter partition size[0b, 0c, 1e, 0.00mb, 0.00gb]: 2600c
partition> label
Ready to label disk, continue? yes

partition> quit

Format all other disks using same method


========================================================================
Create Profile for oracle user

Create or edit /export/home/.bash_profile and add following lines


# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1; export ORACLE_HOME
GRID_HOME=/u01/11.2.0/grid; export GRID_HOME
ORACLE_SID=RACDB1; export ORACLE_SID
#PATH=$ORACLE_HOME/bin:$PATH; export PATH
PATH=$PATH:/usr/local/bin:/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:$ORACLE_HOME/bin

===========================================================

Clone to RACNODE2

Shutdown RACNODE1 Virtual Machine and in VMBox main window 

1. Machines > Clone
2. Add name for new machine (RACNODE2 in my case)
3. On Prompt click on full clone
4. When cloning is finished remove ASM Disks from VMBox Sorage and re attach ASM disks in D:\VMBOX\ASM (asmdisk1.vdi  asmdisk2.vdi  asmdisk3.vdi  asmdisk4.vdi )

=================================================================
Start newly created / cloned RACNODE2 and edit /export/home/oracle/.bash_profile
Change ORACLE_SID parameter from RACDB1 to RACDB2 (Your database instance name)


========================================================================
Create or edit /export/home/.bash_profile and add following lines


# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1; export ORACLE_HOME
GRID_HOME=/u01/11.2.0/grid; export GRID_HOME
ORACLE_SID=RACDB2; export ORACLE_SID
#PATH=$ORACLE_HOME/bin:$PATH; export PATH
PATH=$PATH:/usr/local/bin:/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:$ORACLE_HOME/bin



===========================================================
Change Network information for RACNODE2

Turn on RACNODE2 and change network IPs and Hostname

• update following files to change nodenames

# gedit /etc/nodename

change racnode1 to racnode2

# gedit /etc/hostname.e1000g0

change racnode1 to racnode2

# gedit /etc/hostname.e1000g1

change racnode1-priv to racnode2-priv

• UnPlumb and Plumb again both interfaces

# ifconfig -a unplumb

# ifconfig -a plumb

• Add ips and netmask information for both network interfaces e1000g0 and e1000g1

bash-3.2# ifconfig e1000g0 192.168.0.2 netmask 255.255.255.0 up
bash-3.2# ifconfig e1000g1 10.10.10.2 netmask 255.255.255.0 up

Check that new IPs are assigned to both interfaces

bash-3.2# ifconfig -a
e1000g0: flags=1000843 mtu 1500 index 4
        inet 192.168.1.2 netmask ffffff00 broadcast 192.168.0.255
        ether 0:c:29:8:11:e5
e1000g1: flags=1000843 mtu 1500 index 5
        inet 192.168.2.2 netmask ffffff00 broadcast 10.10.10.255
        ether 0:c:29:8:11:ef
bash-3.2# reboot

=================================================================
Install GI

Post install GI

Run these scripts on both nodes (First run both on First Node and then on Second Node)

 /u01/app/oraInventory/orainstRoot.sh
 /u01/11.2.0/grid/root.sh

=================================================================

# /u01/11.2.0/grid/bin/crs_stat -v

Name           Type           Target    State     Host        
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    racnode1    
ora....ER.lsnr ora....er.type ONLINE    ONLINE    racnode1    
ora....N1.lsnr ora....er.type ONLINE    ONLINE    racnode1    
ora.asm        ora.asm.type   ONLINE    ONLINE    racnode1    
ora.eons       ora.eons.type  ONLINE    ONLINE    racnode1    
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type ONLINE    ONLINE    racnode1    
ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE               
ora.ons        ora.ons.type   ONLINE    ONLINE    racnode1    
ora....SM1.asm application    ONLINE    ONLINE    racnode1    
ora....E1.lsnr application    ONLINE    ONLINE    racnode1    
ora....de1.gsd application    OFFLINE   OFFLINE               
ora....de1.ons application    ONLINE    ONLINE    racnode1    
ora....de1.vip ora....t1.type ONLINE    ONLINE    racnode1    
ora....SM2.asm application    ONLINE    ONLINE    racnode2    
ora....E2.lsnr application    ONLINE    ONLINE    racnode2    
ora....de2.gsd application    OFFLINE   OFFLINE               
ora....de2.ons application    ONLINE    ONLINE    racnode2    
ora....de2.vip ora....t1.type ONLINE    ONLINE    racnode2    
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    racnode1

================================================================

Database Creation