Oracle RAC Management at a Glance

Cluster Verify Utility Usage

./runcluvfy.sh comp nodereach –n raclinux1,raclinux2 –verbose

./runcluvfy.sh stage -post hwos -n raclinux1,raclinux2 –verbose

./runcluvfy.sh stage -pre crsinst -n raclinux1,raclinux2 –verbose

./runcluvfy.sh stage -pre crsinst -n raclinux1,raclinux2 –fixup – fixupdir /tmp/fixups.sh

cluvfy stage –post crsinst –n raclinux1,raclinux2 –verbose

cluvfy stage –pre dbinst –n raclinux1,raclinux2 –verbose

crsctl quety css votedisk

crsctl query crs activeversion

crs_stat –t –v

ps –ef |grep init.d

ps –ef |grep d.bin

11gR2

./crsctl check cluster –all

olsnodes –n

./crsctl query css votedisk

crsctl query crs activeversion

crs_stat –t –v

ps –ef |grep init.d

Removing a successful Cluster configuration

Firstly, stop any active databases running on the node, followed by an ASM instance. You could use the SRVCTL STOP DATABASE/ASM or the SQLPLUS command SHUTDOWN IMMEDIATE to stop the databases and the ASM instance.

$GRID_HOME/crs/install

./rootcrs.pl -verbose –deconfig –force --(except on the last node)

./rootcrs.pl –verbose –deconfig –force –lastnode  --(just on the last node)

./$GRID_HOME/deinstall/deinstall

Removing a Failed Cluster configuration

/$GRID_HOME/crs/install/rootcrs.pl –verbose –deconfig –force --(except on the last node)

/$GRID_HOME/crs/install/rootcrs.pl –verbose –deconfig –force –lastnode --(just on the last node only)

/$GRID_HOME/root.sh

ASM instance creation

In 11gR1 run dbca

In 11gR2 asmca

V$ASM_DISK

V$ASM_DISKGROUP

V$ASM_OPERATION

V$ASM_DISK_STAT

V$ASM_DISKGROUP_STAT

V$ASM_CLIENT

Stopping an ASM instance

srvctl stop asm –n raclinux1 –o normal:immediate:transactional:abort

export ORACLE_SID=+ASM1

export ORACLE_HOME=$ASM_HOME (in 11gR1)

export ORACLE_HOME=$ORA_GRID_HOME (in 11gR2)

sqlplus / as sysasm

SQL> SHUTDOWN NORMAL:IMMEDIATE:TRANSACTIONAL:ABORT

Starting an ASM instance

srvctl start asm –n raclinux1 –o nomount:mount:restrict

export ORACLE_SID=+ASM1

export ORACLE_HOME=$ASM_HOME (in 11gR1)

export ORACLE_HOME=$ORA_GRID_HOME (in 11gR2)

sqlplus / as sysasm

SQL> STARTUP NOMOUNT:MOUNT:RESTRICT

Creating an ASM Disk Group

export ORACLE_SID=+ASM1

export ORACLE_HOME=$ASM_HOME (in 11gR1)

export ORACLE_HOME=$ORA_GRID_HOME (in 11gR2)

SQL> SELECT path,header_status,mode_status,total_mb

FROM v$asm_disk

WHERE header_status IN ('CANDIDATE','FORMER','PROVISIONED');

With External redundancy

CREATE DISKGROUP data EXTERNAL REDUNDANCY DISK '/dev/sde1';

With Normal Redundancy and Fail Group

CREATE DISKGROUP data NORMAL REDUNDANCY

FAILGROUP fgp1 DISK '/dev/sde1',

FAILGROUP fgp2 DISK '/dev/sdf1';

High Redundancy and 3 Fail Groups

The following SQL statement creates a new disk group named DATA with three failure groups with one disk to each failure group, using the Oracle-provided highest level of redundancy, a three-way mirroring-level option:

CREATE DISKGROUP data HIGH REDUNDANCY

FAILGROUP fgp1 DISK '/dev/sde1',

FAILGROUP fgp2 DISK '/dev/sdf1',

FAILGROUP fgp3 DISK '/dev/sdg1';

Altering a disk group

ALTER DISKGROUP data ADD DISK '/dev/sdf1';

ALTER DISKGROUP data ADD DISK '/dev/sdf1','/dev/sdg2'

REBALANCE POWER 3;

Droping a Disk from Disk Group

export ORACLE_SID=+ASM1 – assuming that we are logging in first ASM instance.

export ORACLE_HOME=$ASM_HOME (in 11gR1)

export ORACLE_HOME=$ORA_GRID_HOME (in 11gR2)

Query Disk Group from Sql

SQL> SELECT disk_number,name FROM v$asm_disk WHERE group_number = (SELECT group_number FROM v$asm_diskgroup WHERE NAME = 'DATA');

Drop Disks

ALTER DISKGROUP data DROP DISK '/dev/sdf1';

ALTER DISKGROUP data DROP DISK 'DATA_0001'

REBALANCE POWER 3;

Undrop Disks

ALTER DISKGROUP data UNDROP DISKS;

DROP DISKGROUP data;

DROP DISKGROUP data FORCE;

DROP DISKGROUP data INCLUDING CONTENTS

ASMCMD UTILITY

asmcmd –p

ASMCMD [+] > cd DATA

ASMCMD [+DATA] > cd RACDB

ASMCMD [+DATA/RACDB] cd DATAFILE

ASMCMD [+DATA/RACDB/DATAFILE] > ls –lt

ASMCMD [+] > lsct

ASMCMD [+] > lsdg

Preferred Mirroring

+ASM1.ASM_PREFERRED_MIRROR_READ_GROUPS=DATA.FG1

+ASN2.ASM_PREFERRED_MIRROR_READ_GROUPS=DATA.FG2

ASMCMD rold in 11gR2

export ORACLE_SID=+ASM1

export ORACLE_HOME=$ASM_HOME (in 11gR1)

export ORACLE_HOME=$ORA_GRID_HOME (in 11gR2)

sqlplus / as SYSASM

SQL> CREATE USER my_sysasm IDENTIFIED BY password123;

SQL> GRANT sysasm TO my_sysasm;

To show all ASM Configurations

$GRID_HOME/bin/srvctl config asm –a

Creating ASM Disk Group in Silent Mode

export ORACLE_SID=+ASM1

export ORACLE_HOME=$ORA_GRID_HOME

asmca –silent –createDiskGroup –diskGroupName DG_DATA

-disk '/dev/sdf1'

-redundancy EXTERNAL

-compatible.asm='11.2'

-compatible.rdbms='11.2';

Creating an ASM Volume with ASMCMD

ASMCMD [+] > volcreate -G DG_FLASH -s 1G

ASMCMD [+] volinfo -G DG_FLASH advm_vg

SQL> ALTER DISKGROUP dg_flash ADD VOLUME advm_vg SIZE 1G;

SQL> ALTER DISKGROUP dg_flash RESIZE VOLUME advm_vg SIZE 2G;

SQL> ALTER DISKGROUP dg_flash DISABLE VOLUME advm_vg;

SQL> ALTER DISKGROUP dg_flash ENABLE VOLUME advm_vg;

SQL> ALTER DISKGROUP ALL DISABLE VOLUME ALL;

SQL> ALTER DISKGROUP dg_flash DROP VOLUME advm_vg;

SQL> SELECT volume_name,size_mb,state,usage,volume_device,mountpath

FROM v$asm_volume;

SQL> SELECT volume_name,reads,writes,read_errs,bytes_read,bytes_written

FROM v$asm_volume_stat;

Shutdown and Startup of ASM Instance from asmcmd

ASMCMD [+] > shutdown

ASMCMD [+] > shutdown --immediate

ASMCMD [+] > shutdown --abort

ASMCMD> startup

SPFILE BACKUP FROM DISK TO ASM

ASMCMD> spbackup /u01/app/oracle/product/11.2.0/dbs/ spfile+ASM.ora +DG_DATA/bakspfileASM.ora

Dropping a Disk Group

ASMCMD [+] > dropdg -r dg_data

Checking I/O on a disk group

ASMCMD [+] > iostat -G dg_data

Mounting Disk Groups in Different Modes

ASMCMD [+] > mount -f dg_data

ASMCMD [+] > mount --restrict dg_data

ASMCMD [+] > mount –a

Rebalancing a Disk Group

ASMCMD [+] > rebal --power 4 dg_data

Disk Group Meta Data Backup and Restore

ASMCMD [+] > md_backup /tmp/asmbackups/datapbkps -G DATA

Disk group metadata to be backed up: DATA

Current alias directory path: RACDB

Current alias directory path: RACDB/TEMPFILE

Current alias directory path: RACDB/PARAMETERFILE

Current alias directory path: RACDB/ONLINELOG

Current alias directory path: RACDB/CONTROLFILE

Current alias directory path: RACDB/DATAFILE

ASMCMD [+] > md_restore –-full –G data /tmp/asmbackups/databkps

This example re-creates the DATA disk group and its metadata.

ASMCMD [+] > md_restore –-nodg –G data /tmp/asmbackups/databkps

This example re-creates only metadata for the DATA disk group.

ASMCMD [+] > md_restore –-newdg -o 'data:data2' /tmp/asmbackups/databkps

Managing and Troubleshooting Cluster

[oracle@raclinux1 ~]$ ocrcheck

[oracle@raclinux1 ~]$ crsctl query css votedisk

Initialization and shutdown scripts for Clusterware

Oracle Clusterware makes use of many system-level background scripts to shut down and start the cluster environment. For UNIX and LINUX platforms, these scripts originate from the inittab entries located under the /etc/init.d directory location.

These initialization scripts must run before the Oracle Clusterware daemons can start in the 11g Clusterware environment. Within UNIX and LINUX, these initialization scripts are started and stopped by the init daemon. In order for the init daemon to perform these tasks, entries are written to the /etc/inittab configuration file as shown next:

[root@raclinux1 init.d]# cat /etc/inittab

# Run xdm in runlevel 5

x:5:respawn:/etc/X11/prefdm -nodaemon

h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1

h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1

h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1

[root@raclinux1 init.d]# pwd

/etc/init.d

[root@raclinux1 init.d]# ls -l ini*

-r-xr-xr-x 1 root root 1951 Jun 27 2005 init.crs

-r-xr-xr-x 1 root root 4735 Jun 27 2005 init.crsd

-r-xr-xr-x 1 root root 35401 Jun 27 2005 init.cssd

-r-xr-xr-x 1 root root 3197 Jun 27 2005 init.evmd

$ crsctl

Usage: crsctl check crs - checks the viability of the CRS stack

crsctl check cssd - checks the viability of CSS

crsctl check crsd - checks the viability of CRS

crsctl check evmd - checks the viability of EVM

crsctl set css - sets a parameter override

crsctl get css - gets the value of a CSS parameter

crsctl unset css - sets CSS parameter to its default

crsctl query css votedisk - lists the voting disks used by CSS

crsctl add css votedisk - adds a new voting disk

crsctl delete css votedisk - removes a voting disk

crsctl enable crs - enables startup for all CRS daemons

crsctl disable crs - disables startup for all CRS daemons

crsctl start crs - starts all CRS daemons.

crsctl stop crs - stops all CRS daemons. Stops CRS resources in case of cluster.

crsctl start resources - starts CRS resources.

crsctl stop resources - stops CRS resources.

crsctl debug statedump evm - dumps state info for evm objects

crsctl debug statedump crs - dumps state info for crs objects

crsctl debug statedump css - dumps state info for css objects

crsctl debug log css [module:level]{,module:level} ...

- Turns on debugging for CSS

crsctl debug trace css - dumps CSS in-memory tracing cache

crsctl debug log crs [module:level]{,module:level} ...

- Turns on debugging for CRS

crsctl debug trace crs - dumps CRS in-memory tracing cache

crsctl debug log evm [module:level]{,module:level} ...

- Turns on debugging for EVM

crsctl debug trace evm - dumps EVM in-memory tracing cache

crsctl debug log res turns on debugging for resources

crsctl query crs softwareversion [] - lists the version of CRS software installed

crsctl query crs activeversion - lists the CRS software operating version

crsctl lsmodules css - lists the CSS modules that can be used for debugging

crsctl lsmodules crs - lists the CRS modules that can be used for debugging

crsctl lsmodules evm - lists the EVM modules that can be used for debugging

CRSCTL COMMAND

[root@raclinux1 ~]# mkdir -p /11g/votedisk

[root@raclinux1 ~]# chmod 777 //11g/votedisk

[root@raclinux1 ~]# /u01/app/oracle/product/11.1.0/db_1/bin/crsctl add css votedisk /11g/votedisk/vdisk4

Cluster is not in a ready state for online disk addition

[root@raclinux1 ~]# /u01/app/oracle/product/11.1.0/db_1/bin/crsctl stop crs

Stopping resources.

Successfully stopped CRS resources

Stopping CSSD.

Shutting down CSS daemon.

Shutdown request successfully issued.

[root@raclinux1 ~]#

[root@raclinux1 ~]# /u01/app/oracle/product/11.1.0/db_1/bin/crsctl add css votedisk /11g/votedisk/vdisk4 -force

Now formatting voting disk: /11g/votedisk/vdisk4

successful addition of votedisk /11g/votedisk/vdisk4.

[root@raclinux1 ~]# /u01/app/oracle/product/11.1.0/db_1/bin/crsctl add css votedisk /11g/votedisk/vdisk5 -force

Now formatting voting disk: /11g/votedisk/vdisk5

successful addition of votedisk /11g/votedisk/vdisk5.

[root@raclinux1 ~]# /u01/app/oracle/product/11.1.0/db_1/bin/crsctl start crs Attempting to start CRS stack

The CRS stack will be started shortly

[oracle@raclinux1 ~]$ crsctl query css votedisk

$ crs_stat -t

$ crs_stat -p

crs_stat –ls

$ ocrcheck