Oracle Real Application Cluster Managment


Cluster Control utility performs various administrative operation of oracle clusterware. It is located in $ORA_CRS_HOME/bin and must be executed by the “root” user.

a. To check the current state of all oracle clusterware daemon:

[root@rac1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

b.You can also check the state of individual oracle clusterware daemon:

[root@rac1 bin]# ./crsctl check cssd
CSS appears healthy

[root@rac1 bin]# ./crsctl check crsd
CRS appears healthy

[root@rac1 bin]# ./crsctl check evmd
EVM appears healthy

c. To start oracle clusterware

[root@rac1 bin]# ./crsctl start crs

Attempting to start CRS stack
The CRS stack will be started shortly

d. To stop oracle clusterware

[root@rac1 bin]# ./crsctl stop crs
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.

e. To disable  oracle clusterware:

[root@rac1 bin]# ./crsctl disable crs

f.  To enable  oracle clusterware:

[root@rac1 bin]# ./crsctl enable crs

g. To get current value of CSS parameter

[root@rac1 bin]# ./crsctl get css
for  example: to get value of misscount parameter
 [root@rac1 bin]# ./crsctl get css misscount

h. To set a new value of CSS parameter

[root@rac1 bin]# ./crsctl set css
for  example: to set value of misscount parameter

[root@rac1 bin]# ./crsctl set css misscount 120
Configuration parameter misscount is now set to 120.

i. To unset CSS parameter value

[root@rac1 bin]# ./crsctl unset css
for  example: to unset value of misscount parameter

[root@rac1 bin]# ./crsctl unset css misscount
Configuration parameter misscount is now undefined.

j. To list the module for debugging in CSS

[root@rac2 bin]# ./crsctl lsmodules css

The following are the CSS modules ::

 It reports the current state of resources configured in the OCR.

[oracle@rac1 bin]$ ./crs_stat -t
Name                      Type              Target       State         Host
ora….C1.inst             application    ONLINE    ONLINE          rac1
ora….C2.inst             application    ONLINE    ONLINE          rac2
ora….AC1.srv            application    ONLINE    ONLINE          rac1          application    ONLINE    ONLINE          rac1
ora.RAC.db                application    ONLINE    ONLINE         rac2
ora….AC1.srv            application    ONLINE    ONLINE          rac1
ora….ice2.cs             application    ONLINE    ONLINE          rac1
ora….AC1.srv            application    ONLINE    ONLINE          rac1


 This command used to stop resource or cluster member.

[root@rac1 bin]# ./crs_stop ora.rac1.ons
Attempting to stop `ora.rac1.ons` on member `rac1`
Stop of `ora.rac1.ons` on member `rac1` succeeded.


 This command used to start resource or cluster member.

[root@rac1 bin]# ./crs_start ora.rac1.ons
Attempting to start `ora.rac1.ons` on member `rac1`
Start of `ora.rac1.ons` on member `rac1` succeeded.


 It dumps the contents of OCR into a text file.

 [root@rac1 bin]# ./ocrdump /home/oracle/ocr.dmp


 It verifies the integrity of the OCR.

[root@rac2 bin]# ./ocrcheck

Status of Oracle Cluster Registry is as follows :
Version                                :    2
Total space (kbytes)         :    5237072
Used space (kbytes)         :    9360
Available space (kbytes)  :    5227712
ID                                        :    794527192
Device/File Name             :    /apps/oracle/oradata/ocr
Device/File integrity check succeeded
Cluster registry integrity check succeeded


 It perform various administrative operation on the OCR.

Remove Node From Oracle Rac 10g ( Part-2)

 Remove Node From The Database :  ( From The Node You want to Delete) 

**Before You Start Deleting Node From Database You need To update inventory by :

./runInstallerupdateNodeList ORACLE_HOME=<Database home> “CLUSTER_NODES=<node to be removed>” –local

After Perform This You can Start De-install Process Like The Following : 

 You now need to update the corresponding  inventory on the remaining nodes. You can use the following command from the first node:
./runInstallerupdateNodeList ORACLE_HOME=<Database home> “CLUSTER_NODES=<remaining nodes>”

Remove Node From ASM : 

maybe some of us will be conflict with the above steps , I Above Step we Remove ASM from Node that we want to delete But ASM still exists , And On Remaining Node The Deleted Node For ASM still exists So We Need To update the ASM to Delete  Like The Following :

Update Inventory From Node We Want delete : 

./runInstallerupdateNodeList ORACLE_HOME=<ASM home> “CLUSTER_NODES=<node to be removed>” –local

update the corresponding inventory on the remaining nodes. You can use the following command from the first node:
./runInstallerupdateNodeList ORACLE_HOME=<ASM home> “CLUSTER_NODES=<remaining nodes>”

Remove Node From Oracle Clusterware From The First Node Do the Below Steps : 

1-Run This Command :

<Oracle Clusterware home>/bin/racgons remove_config <Node to be removed>:6200 
**6200 : Can Be Changed, And Its Called Remote Port 

<Oracle Clusterware home>/opmn/conf/ons.config 

2-From The Node You want To remove as root User : 

<Oracle Clusterwarehome>/install/

3-From The First Node as Root User : 

<Oracle Clusterware home>/bin/olsnodes –n
<Oracle Clusterware home>/install/ <node name to be deleted>,<node number to be deleted>.


Remove Node From Oracle Cluster-ware (From The Node You want to be Delete)

To Update Inventory Run The Following Command : 


<Oracle Clusterwarehome>/oui/bin/runInstaller  –updateNodeList ORACLE_HOME=<Oracle Clusterwarehome>  “CLUSTER_NODES=<Node to be deleted>”   CRS=TRUE -local

Run Installer — > De-install 

From The First Node after Finish The De-install :
<Oracle Clusterwarehome>/oui/bin/runInstallerupdateNodeList  ORACLE_HOME=<Oracle Clusterware home>  “CLUSTER_NODES=<Remaining nodes>” CRS=TRUE
 Now You Are Delete The Node ..
To Check

 •srvctl  status nodeapps -n <Deleted node>should get a message saying Invalid node.
crs_stat | grep -i <Deleted node>should not get any output.
olsnodes –n should get all the present nodes list without the deleted node.
Thank You
Osama  Mustafa 

Remove Node From Oracle RAC 10g (Part-1)

In This Article I Will Show You How to Remove Node From Oracle Real Application Cluster 10g :

Let’s Start : 

Main Steps To Remove Node From Oracle Real Application Cluster : 

1.Delete the instance on the node to be deleted.
2.Clean up the ASM instance.
3.Remove the listener from the node to be deleted.
4.Remove the node from the database.
5.Remove the node from ASM.
6.Remove ONS configuration from the node to be deleted.
7.Remove the node from the cluster-ware.
 **The First Steps Is To Delete Instance On The Node (From The Node you want to delete) —- >  DBCA
On The Above Message Press Ok .
**Clean Up The ASM Instance : (From node you want to delete) 

you need to manually remove the initialization parameter file of that ASM instance
remove all the log files of that ASM instance. These files are generally located in the $ORACLE_BASE/admin directory then remove the associated ASM entry from the /etc/oratabfile
Remove The Listener (From the Node You want to delete )——> NETCA

Note : You need to choose the Node you want to be deleted , Not Select All .

Part Two In Separate Article 

Oracle Rac 10g Add Node Steps

This Article Discuss Adding/Remove Node In Oracle Real application Cluster 10g The below Steps easy all you have to do is follow up with them Let’s Start :


1-Check Pre-requisities On All Node , This Include Old Node And New Node .

cluvfy  stage -pre crsinst -n -r 10gR2

 2-From the First Node , Execute the following Command :


The Final Result For Step 2 .

3-We need to configure new ONS (From The First Node) :

cd cluster_home/bin
cat cluster_home/opmn/conf/ons.config (to get the remoteport)
./racgons add_config Node-name:6200 (remoteport)

4-Optional : needed Only If Specific Home Directory to host ASM (From the First Node)


5- From The First node we need to Add RAC

cd cluster_home/bin


6-Add Listener , Do This Steps From The New Node On Terminal

netca –> Cluster Configuration —>Name of New Node

7-Final Step , Add Database Instance To new node ( from the First Node ) Using

DBCA —> Oracle Real Application Cluster —> Instance Management —> Add Instance



You Need Follow Up with Screen as Normal Installation Of Database
List of cluster your RAC database and enter SYS credentials 
List of cluster database instances , next
Instance naming and node selection . Instance Name
if you are using ASM for your database storage,the DBCA detects the need for an ASM instance creation on the new nodeThis must be done before the DBCA can create the database instance on that node. Click Yes.

Thank You
Osama Mustafa

I will Talk Later How To Remove Node From Oracle Real Application Cluster .

Change VIP Addresses

1.Determine the interface used to support your VIP:
 $ ifconfig -a
2.Stop all resources depending on the VIP:
$ srvctl stop instance -d DB -i DB1
$ srvctl stop asm -n node1
# srvctl stop nodeapps -n node1

3.Verify that the VIP is no longer running:

$ ifconfig -a
$ crs_stat
4.Change IP in /etc/hostsand DNS.

5.Modify your VIP address using srvctl:
# srvctl modify nodeapps -n node1 -A
6.Start nodeappsand all resources depending on it:
# srvctl start nodeapps -n node1
7.Repeat from step 1 for the next node.
Thank You 
Osama Mustafa 




UNKNOWN State Of RAC Resources

Sometimes when you check Oracle Cluster Status Via “crs_stat -t” :

ora.orcl.db application ONLINE ONLINE rac1
ora....11.inst application ONLINE ONLINE rac1
ora....SM1.asm application ONLINE ONLINE rac1
ora....DC.lsnr application ONLINE ONLINE rac1
ora....idc.gsd application ONLINE ONLINE rac1
ora....idc.ons application ONLINE ONLINE rac1 application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE UNKNOWN rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora....dc2.gsd application ONLINE ONLINE rac2
ora....dc2.ons application ONLINE ONLINE rac2 application ONLINE ONLINE rac2

If You try to restart Oracle Cluster via

./crs_stop -all
./crs_start -all

It will not working so you have to do the following simple steps :

 First :

crs_stop -f Service_name
Example :
crs_stop -f ora.rac2.ASM2.asm 

 Second :

crs_start -f services_name

Example :

crs_start -f ora.rac2.ASM2.asm 

Thank you