Oracle Real Application Cluster Managment


Cluster Control utility performs various administrative operation of oracle clusterware. It is located in $ORA_CRS_HOME/bin and must be executed by the “root” user.

a. To check the current state of all oracle clusterware daemon:

[root@rac1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

b.You can also check the state of individual oracle clusterware daemon:

[root@rac1 bin]# ./crsctl check cssd
CSS appears healthy

[root@rac1 bin]# ./crsctl check crsd
CRS appears healthy

[root@rac1 bin]# ./crsctl check evmd
EVM appears healthy

c. To start oracle clusterware

[root@rac1 bin]# ./crsctl start crs

Attempting to start CRS stack
The CRS stack will be started shortly

d. To stop oracle clusterware

[root@rac1 bin]# ./crsctl stop crs
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.

e. To disable  oracle clusterware:

[root@rac1 bin]# ./crsctl disable crs

f.  To enable  oracle clusterware:

[root@rac1 bin]# ./crsctl enable crs

g. To get current value of CSS parameter

[root@rac1 bin]# ./crsctl get css
for  example: to get value of misscount parameter
 [root@rac1 bin]# ./crsctl get css misscount

h. To set a new value of CSS parameter

[root@rac1 bin]# ./crsctl set css
for  example: to set value of misscount parameter

[root@rac1 bin]# ./crsctl set css misscount 120
Configuration parameter misscount is now set to 120.

i. To unset CSS parameter value

[root@rac1 bin]# ./crsctl unset css
for  example: to unset value of misscount parameter

[root@rac1 bin]# ./crsctl unset css misscount
Configuration parameter misscount is now undefined.

j. To list the module for debugging in CSS

[root@rac2 bin]# ./crsctl lsmodules css

The following are the CSS modules ::

 It reports the current state of resources configured in the OCR.

[oracle@rac1 bin]$ ./crs_stat -t
Name                      Type              Target       State         Host
ora….C1.inst             application    ONLINE    ONLINE          rac1
ora….C2.inst             application    ONLINE    ONLINE          rac2
ora….AC1.srv            application    ONLINE    ONLINE          rac1          application    ONLINE    ONLINE          rac1
ora.RAC.db                application    ONLINE    ONLINE         rac2
ora….AC1.srv            application    ONLINE    ONLINE          rac1
ora….ice2.cs             application    ONLINE    ONLINE          rac1
ora….AC1.srv            application    ONLINE    ONLINE          rac1


 This command used to stop resource or cluster member.

[root@rac1 bin]# ./crs_stop ora.rac1.ons
Attempting to stop `ora.rac1.ons` on member `rac1`
Stop of `ora.rac1.ons` on member `rac1` succeeded.


 This command used to start resource or cluster member.

[root@rac1 bin]# ./crs_start ora.rac1.ons
Attempting to start `ora.rac1.ons` on member `rac1`
Start of `ora.rac1.ons` on member `rac1` succeeded.


 It dumps the contents of OCR into a text file.

 [root@rac1 bin]# ./ocrdump /home/oracle/ocr.dmp


 It verifies the integrity of the OCR.

[root@rac2 bin]# ./ocrcheck

Status of Oracle Cluster Registry is as follows :
Version                                :    2
Total space (kbytes)         :    5237072
Used space (kbytes)         :    9360
Available space (kbytes)  :    5227712
ID                                        :    794527192
Device/File Name             :    /apps/oracle/oradata/ocr
Device/File integrity check succeeded
Cluster registry integrity check succeeded


 It perform various administrative operation on the OCR.

PRVF-5436 PRVF-9652 Cluster Time Synchronization Services check failed

$GRID_HOME/bin/cluvfy comp clocksync
Cause of the Problem
The problem happened because the time service (such as NTP) is not running or doesn’t have slew option set.

Solution of the Problem
The problem happened because time service NTP is not running or doesn’t have slew option set.
So, ensure that NTP service is up and running.
Based on the operating system solutions are given below.

1) Linux:

To verify service as root user issue,
# /sbin/service ntpd status
ntpd (pid 4423) is running…

Check process is running or not by.
# ps -ef|grep ntp
ntp 4209 1 0 Mar10 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ -x

# grep OPTIONS /etc/sysconfig/ntpd
OPTIONS=”-u ntp:ntp -p /var/run/ -x”

If NTP service is not started then to start the service issue,
To start the service issue,

# /sbin/service ntpd start

2) Solaris:

To verify the service issue,

# /usr/bin/svcs ntp
online 3:29:11 svc:/network/ntp:default

# ps -ef|grep ntp
root 21223 1 0 Mar 10 ? 0:21 /usr/lib/inet/xntpd

# grep slewalways /etc/inet/ntp.conf
slewalways yes

To start the NTP service issue,

# /usr/sbin/svcadm enable ntp

3) HP-UX:

To verify the service issue,

# ps -ef|grep ntp
root 6022 1 0 14:23:42 ? 0:01 /usr/sbin/xntpd -x

# grep XNTPD_ARGS /etc/rc.config.d/netdaemons
export XNTPD_ARGS=”-x”

To start the service issue,
# /sbin/init.d/xntpd start

4) AIX:

To verify the service issue,

# /usr/bin/lssrc -ls xntpd
xntpd tcpip 368754 active

# ps -ef|grep ntp
root 786614 151686 0 08:02:32 – 0:00 /usr/sbin/xntpd -x

# grep xntpd /etc/rc.tcpip
start /usr/sbin/xntpd “$src_running” -a “-x”

To start the service issue,

# /usr/bin/startsrc -s xntpd -a “-x”

Thank you
Osama mustafa

CRS-4640 Error on Starting 11gR2 clusterware

[root@RAC01 cssd]# /oragrid/product/11.2/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
When Trying To start Oracle Cluster with following Command :
[root@RAC01 cssd]# /oragrid/product/11.2/bin/crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

When Trying To Stop Oracle Cluster With Following Command : 

crsctl stop crs command failed

[root@RAC01 cssd]# /oragrid/product/11.2/bin/crsctl stop crs
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.

Solution is Simple :
Just Run 
[root@RAC01 cssd]# /oragrid/product/11.2/bin/crsctl start cluster 

[root@RAC01 ~]# /oragrid/product/11.2/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

Thank you
Osama mustafa

Remove Node From Oracle Rac 10g ( Part-2)

 Remove Node From The Database :  ( From The Node You want to Delete) 

**Before You Start Deleting Node From Database You need To update inventory by :

./runInstallerupdateNodeList ORACLE_HOME=<Database home> “CLUSTER_NODES=<node to be removed>” –local

After Perform This You can Start De-install Process Like The Following : 

 You now need to update the corresponding  inventory on the remaining nodes. You can use the following command from the first node:
./runInstallerupdateNodeList ORACLE_HOME=<Database home> “CLUSTER_NODES=<remaining nodes>”

Remove Node From ASM : 

maybe some of us will be conflict with the above steps , I Above Step we Remove ASM from Node that we want to delete But ASM still exists , And On Remaining Node The Deleted Node For ASM still exists So We Need To update the ASM to Delete  Like The Following :

Update Inventory From Node We Want delete : 

./runInstallerupdateNodeList ORACLE_HOME=<ASM home> “CLUSTER_NODES=<node to be removed>” –local

update the corresponding inventory on the remaining nodes. You can use the following command from the first node:
./runInstallerupdateNodeList ORACLE_HOME=<ASM home> “CLUSTER_NODES=<remaining nodes>”

Remove Node From Oracle Clusterware From The First Node Do the Below Steps : 

1-Run This Command :

<Oracle Clusterware home>/bin/racgons remove_config <Node to be removed>:6200 
**6200 : Can Be Changed, And Its Called Remote Port 

<Oracle Clusterware home>/opmn/conf/ons.config 

2-From The Node You want To remove as root User : 

<Oracle Clusterwarehome>/install/

3-From The First Node as Root User : 

<Oracle Clusterware home>/bin/olsnodes –n
<Oracle Clusterware home>/install/ <node name to be deleted>,<node number to be deleted>.


Remove Node From Oracle Cluster-ware (From The Node You want to be Delete)

To Update Inventory Run The Following Command : 


<Oracle Clusterwarehome>/oui/bin/runInstaller  –updateNodeList ORACLE_HOME=<Oracle Clusterwarehome>  “CLUSTER_NODES=<Node to be deleted>”   CRS=TRUE -local

Run Installer — > De-install 

From The First Node after Finish The De-install :
<Oracle Clusterwarehome>/oui/bin/runInstallerupdateNodeList  ORACLE_HOME=<Oracle Clusterware home>  “CLUSTER_NODES=<Remaining nodes>” CRS=TRUE
 Now You Are Delete The Node ..
To Check

 •srvctl  status nodeapps -n <Deleted node>should get a message saying Invalid node.
crs_stat | grep -i <Deleted node>should not get any output.
olsnodes –n should get all the present nodes list without the deleted node.
Thank You
Osama  Mustafa 

Remove Node From Oracle RAC 10g (Part-1)

In This Article I Will Show You How to Remove Node From Oracle Real Application Cluster 10g :

Let’s Start : 

Main Steps To Remove Node From Oracle Real Application Cluster : 

1.Delete the instance on the node to be deleted.
2.Clean up the ASM instance.
3.Remove the listener from the node to be deleted.
4.Remove the node from the database.
5.Remove the node from ASM.
6.Remove ONS configuration from the node to be deleted.
7.Remove the node from the cluster-ware.
 **The First Steps Is To Delete Instance On The Node (From The Node you want to delete) —- >  DBCA
On The Above Message Press Ok .
**Clean Up The ASM Instance : (From node you want to delete) 

you need to manually remove the initialization parameter file of that ASM instance
remove all the log files of that ASM instance. These files are generally located in the $ORACLE_BASE/admin directory then remove the associated ASM entry from the /etc/oratabfile
Remove The Listener (From the Node You want to delete )——> NETCA

Note : You need to choose the Node you want to be deleted , Not Select All .

Part Two In Separate Article 

Oracle Rac 10g Add Node Steps

This Article Discuss Adding/Remove Node In Oracle Real application Cluster 10g The below Steps easy all you have to do is follow up with them Let’s Start :


1-Check Pre-requisities On All Node , This Include Old Node And New Node .

cluvfy  stage -pre crsinst -n -r 10gR2

 2-From the First Node , Execute the following Command :


The Final Result For Step 2 .

3-We need to configure new ONS (From The First Node) :

cd cluster_home/bin
cat cluster_home/opmn/conf/ons.config (to get the remoteport)
./racgons add_config Node-name:6200 (remoteport)

4-Optional : needed Only If Specific Home Directory to host ASM (From the First Node)


5- From The First node we need to Add RAC

cd cluster_home/bin


6-Add Listener , Do This Steps From The New Node On Terminal

netca –> Cluster Configuration —>Name of New Node

7-Final Step , Add Database Instance To new node ( from the First Node ) Using

DBCA —> Oracle Real Application Cluster —> Instance Management —> Add Instance



You Need Follow Up with Screen as Normal Installation Of Database
List of cluster your RAC database and enter SYS credentials 
List of cluster database instances , next
Instance naming and node selection . Instance Name
if you are using ASM for your database storage,the DBCA detects the need for an ASM instance creation on the new nodeThis must be done before the DBCA can create the database instance on that node. Click Yes.

Thank You
Osama Mustafa

I will Talk Later How To Remove Node From Oracle Real Application Cluster .

Change VIP Addresses

1.Determine the interface used to support your VIP:
 $ ifconfig -a
2.Stop all resources depending on the VIP:
$ srvctl stop instance -d DB -i DB1
$ srvctl stop asm -n node1
# srvctl stop nodeapps -n node1

3.Verify that the VIP is no longer running:

$ ifconfig -a
$ crs_stat
4.Change IP in /etc/hostsand DNS.

5.Modify your VIP address using srvctl:
# srvctl modify nodeapps -n node1 -A
6.Start nodeappsand all resources depending on it:
# srvctl start nodeapps -n node1
7.Repeat from step 1 for the next node.
Thank You 
Osama Mustafa 




Some Basic To backup Voting Disk And OCR For Oracle RAC

We Need To Backup Voting Disk and OCR For Oracle RAC For Disaster Situation :

Add Voting Disk :

# crsctl add css votedisk

To remove a voting disk:

# crsctl delete css votedisk

If Oracle Clusterware is down on all nodes, use the –force option:

# crsctl add css votedisk -force
# crsctl delete css votedisk -force

 Backup Voting Disk :

$ crsctl query css votedisk
$ dd if= of= bs=4k


Oracle RAC Taking Backup For OCR By Default , You will Find It On :

 cd $ORACLE_BASE/Crs/cdata/jfv_clus

Change the default automatic backup location:

# ocrconfig –backuploc /shared/bak

Back Up OCR Manually :

 # ocrconfig –export file name

 Recover OCR Using Physical Backups:

1.Locate a physical backup:
 $ ocrconfigshowbackup
2.Review its contents:

# ocrdumpbackupfile file_name
 3.Stop Oracle Clusterwareon all nodes:

 # crsctl stop crs
4.Restore the physical OCR backup:

# ocrconfig –restore <CRS HOME>/cdata/jfv_clus/day.ocr

5.Restart Oracle Clusterware on all nodes:

 # crsctl start crs
6.Check OCR integrity:

$ cluvfy comp ocr -n all

Thank You
Osama Mustafa 


UNKNOWN State Of RAC Resources

Sometimes when you check Oracle Cluster Status Via “crs_stat -t” :

ora.orcl.db application ONLINE ONLINE rac1
ora....11.inst application ONLINE ONLINE rac1
ora....SM1.asm application ONLINE ONLINE rac1
ora....DC.lsnr application ONLINE ONLINE rac1
ora....idc.gsd application ONLINE ONLINE rac1
ora....idc.ons application ONLINE ONLINE rac1 application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE UNKNOWN rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora....dc2.gsd application ONLINE ONLINE rac2
ora....dc2.ons application ONLINE ONLINE rac2 application ONLINE ONLINE rac2

If You try to restart Oracle Cluster via

./crs_stop -all
./crs_start -all

It will not working so you have to do the following simple steps :

 First :

crs_stop -f Service_name
Example :
crs_stop -f ora.rac2.ASM2.asm 

 Second :

crs_start -f services_name

Example :

crs_start -f ora.rac2.ASM2.asm 

Thank you

Create Enterprise manager for RAC (Real Application Cluster)

Hi All ,

Sometimes we need to create enterprise manager for Rac Environment , its the same way for One instance with different Command . 

We are here talk about 10g , 11g database .
you follow the below steps to create or recreate enterprise manager LET’S START :

if you already have RAC – enterprise manager and you need to recreate then start from here

**Stop Dbconsole for the both nodes :

Commands :

On Node-1
emctl stop dbconsole .
On Node 2 :
emctl stop dbconsole .

**Drop dbconsole with re posterity from node 1 :

Command :

$emca -deconfig dbcontrol db -repos drop -cluster

“you will wait for while until repos been dropped”.

**recreate dbconsole with re posterity from node 1 “if you don’t have EM for your Rac start from here”
Command :

$emca -config dbcontrol db -repos create -cluster 

“follow the instsruction and fill the information such as DB SID,Listener Port , Crs Name”

To know your CRS NAME :
$CRS_HOME/bin/cemutlo -n

the output will be your crs name .

Thank you
Osama mustafa