CRS – Technology Geek

Oracle Real Application Cluster Managment

Posted on June 5, 2012December 23, 2018 by Osama Mustafa in Database

1. CRSCTL:

Cluster Control utility performs various administrative operation of oracle clusterware. It is located in $ORA_CRS_HOME/bin and must be executed by the “root” user.

a. To check the current state of all oracle clusterware daemon:

[root@rac1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

b.You can also check the state of individual oracle clusterware daemon:

[root@rac1 bin]# ./crsctl check cssd
CSS appears healthy

[root@rac1 bin]# ./crsctl check crsd
CRS appears healthy

[root@rac1 bin]# ./crsctl check evmd
EVM appears healthy

c. To start oracle clusterware

[root@rac1 bin]# ./crsctl start crs

Attempting to start CRS stack
The CRS stack will be started shortly

d. To stop oracle clusterware

[root@rac1 bin]# ./crsctl stop crs
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.

e. To disable oracle clusterware:

[root@rac1 bin]# ./crsctl disable crs

f. To enable oracle clusterware:

[root@rac1 bin]# ./crsctl enable crs

g. To get current value of CSS parameter

[root@rac1 bin]# ./crsctl get css
for example: to get value of misscount parameter
[root@rac1 bin]# ./crsctl get css misscount
60

h. To set a new value of CSS parameter

[root@rac1 bin]# ./crsctl set css
for example: to set value of misscount parameter

[root@rac1 bin]# ./crsctl set css misscount 120
Configuration parameter misscount is now set to 120.

i. To unset CSS parameter value

[root@rac1 bin]# ./crsctl unset css
for example: to unset value of misscount parameter

[root@rac1 bin]# ./crsctl unset css misscount
Configuration parameter misscount is now undefined.

j. To list the module for debugging in CSS

[root@rac2 bin]# ./crsctl lsmodules css

The following are the CSS modules ::
CSSD
COMMCRS
COMMNS

2. CRS_STAT:
It reports the current state of resources configured in the OCR.

[oracle@rac1 bin]$ ./crs_stat -t
Name          Type              Target       State         Host
———————————————————————————–
ora….C1.inst             application    ONLINE    ONLINE          rac1
ora….C2.inst             application    ONLINE    ONLINE          rac2
ora….AC1.srv            application    ONLINE    ONLINE          rac1
ora.RAC.abc.cs          application    ONLINE    ONLINE          rac1
ora.RAC.db                application    ONLINE    ONLINE         rac2
ora….AC1.srv            application    ONLINE    ONLINE         rac1
ora….ice2.cs             application    ONLINE    ONLINE          rac1
ora….AC1.srv            application    ONLINE    ONLINE          rac1

3. CRS_STOP:

This command used to stop resource or cluster member.

[root@rac1 bin]# ./crs_stop ora.rac1.ons
Attempting to stop `ora.rac1.ons` on member `rac1`
Stop of `ora.rac1.ons` on member `rac1` succeeded.

4. CRS_START:

This command used to start resource or cluster member.

[root@rac1 bin]# ./crs_start ora.rac1.ons
Attempting to start `ora.rac1.ons` on member `rac1`
Start of `ora.rac1.ons` on member `rac1` succeeded.

5. OCRDUMP :

It dumps the contents of OCR into a text file.

[root@rac1 bin]# ./ocrdump /home/oracle/ocr.dmp

6. OCRCHECK :

It verifies the integrity of the OCR.

[root@rac2 bin]# ./ocrcheck

Status of Oracle Cluster Registry is as follows :
Version                    :    2
Total space (kbytes)         :    5237072
Used space (kbytes)         :    9360
Available space (kbytes) :    5227712
ID                      : 794527192
Device/File Name       :    /apps/oracle/oradata/ocr
Device/File integrity check succeeded
Cluster registry integrity check succeeded

7. OCRCONFIG:

It perform various administrative operation on the OCR.

PRVF-5436 PRVF-9652 Cluster Time Synchronization Services check failed

Posted on May 28, 2012December 23, 2018 by Osama Mustafa in Database

$GRID_HOME/bin/cluvfy comp clocksync
Cause of the Problem
The problem happened because the time service (such as NTP) is not running or doesn’t have slew option set.

Solution of the Problem
The problem happened because time service NTP is not running or doesn’t have slew option set.
So, ensure that NTP service is up and running.
Based on the operating system solutions are given below.

1) Linux:

To verify service as root user issue,
# /sbin/service ntpd status
ntpd (pid 4423) is running…

Check process is running or not by.
# ps -ef|grep ntp
ntp 4209 1 0 Mar10 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -x

# grep OPTIONS /etc/sysconfig/ntpd
OPTIONS=”-u ntp:ntp -p /var/run/ntpd.pid -x”

If NTP service is not started then to start the service issue,
To start the service issue,

# /sbin/service ntpd start

2) Solaris:

To verify the service issue,

# /usr/bin/svcs ntp
STATE STIME FMRI
online 3:29:11 svc:/network/ntp:default

# ps -ef|grep ntp
root 21223 1 0 Mar 10 ? 0:21 /usr/lib/inet/xntpd

# grep slewalways /etc/inet/ntp.conf
slewalways yes

To start the NTP service issue,

# /usr/sbin/svcadm enable ntp

3) HP-UX:

To verify the service issue,

# ps -ef|grep ntp
root 6022 1 0 14:23:42 ? 0:01 /usr/sbin/xntpd -x

# grep XNTPD_ARGS /etc/rc.config.d/netdaemons
export XNTPD_ARGS=”-x”

To start the service issue,
# /sbin/init.d/xntpd start

4) AIX:

To verify the service issue,

# /usr/bin/lssrc -ls xntpd
xntpd tcpip 368754 active

# ps -ef|grep ntp
root 786614 151686 0 08:02:32 – 0:00 /usr/sbin/xntpd -x

# grep xntpd /etc/rc.tcpip
start /usr/sbin/xntpd “$src_running” -a “-x”

To start the service issue,

# /usr/bin/startsrc -s xntpd -a “-x”

Thank you
Osama mustafa

CRS-4640 Error on Starting 11gR2 clusterware

Posted on May 24, 2012December 23, 2018 by Osama Mustafa in Database

ERROR :

[root@RAC01 cssd]# /oragrid/product/11.2/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager

When Trying To start Oracle Cluster with following Command :

[root@RAC01 cssd]# /oragrid/product/11.2/bin/crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

When Trying To Stop Oracle Cluster With Following Command :

crsctl stop crs command failed

[root@RAC01 cssd]# /oragrid/product/11.2/bin/crsctl stop crs
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.

Solution is Simple :

Just Run

[root@RAC01 cssd]# /oragrid/product/11.2/bin/crsctl start cluster

[root@RAC01 ~]# /oragrid/product/11.2/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

Thank you
Osama mustafa

Remove Node From Oracle Rac 10g ( Part-2)

Posted on April 17, 2012December 23, 2018 by Osama Mustafa in Database

Remove Node From The Database : ( From The Node You want to Delete)

**Before You Start Deleting Node From Database You need To update inventory by :

./runInstaller –updateNodeList ORACLE_HOME=<Database home> “CLUSTER_NODES=<node to be removed>” –local

After Perform This You can Start De-install Process Like The Following :

You now need to update the corresponding inventory on the remaining nodes. You can use the following command from the first node:

./runInstaller –updateNodeList ORACLE_HOME=<Database home> “CLUSTER_NODES=<remaining nodes>”

Remove Node From ASM :

maybe some of us will be conflict with the above steps , I Above Step we Remove ASM from Node that we want to delete But ASM still exists , And On Remaining Node The Deleted Node For ASM still exists So We Need To update the ASM to Delete Like The Following :

Update Inventory From Node We Want delete :

./runInstaller –updateNodeList ORACLE_HOME=<ASM home> “CLUSTER_NODES=<node to be removed>” –local

update the corresponding inventory on the remaining nodes. You can use the following command from the first node:

./runInstaller –updateNodeList ORACLE_HOME=<ASM home> “CLUSTER_NODES=<remaining nodes>”

Remove Node From Oracle Clusterware From The First Node Do the Below Steps :

1-Run This Command :

<Oracle Clusterware home>/bin/racgons remove_config <Node to be removed>:6200

**6200 : Can Be Changed, And Its Called Remote Port

<Oracle Clusterware home>/opmn/conf/ons.config

2-From The Node You want To remove as root User :

<Oracle Clusterwarehome>/install/rootdelete.sh

3-From The First Node as Root User :

<Oracle Clusterware home>/bin/olsnodes –n

Then

<Oracle Clusterware home>/install/rootdeletenode.sh <node name to be deleted>,<node number to be deleted>.

Remove Node From Oracle Cluster-ware (From The Node You want to be Delete)

To Update Inventory Run The Following Command :

<Oracle Clusterwarehome>/oui/bin/runInstaller –updateNodeList ORACLE_HOME=<Oracle Clusterwarehome> “CLUSTER_NODES=<Node to be deleted>” CRS=TRUE -local

Run Installer — > De-install

From The First Node after Finish The De-install :

<Oracle Clusterwarehome>/oui/bin/runInstaller –updateNodeList ORACLE_HOME=<Oracle Clusterware home> “CLUSTER_NODES=<Remaining nodes>” CRS=TRUE

Now You Are Delete The Node ..

To Check

•srvctl status nodeapps -n <Deleted node>should get a message saying Invalid node.

•crs_stat | grep -i <Deleted node>should not get any output.

•olsnodes –n should get all the present nodes list without the deleted node.

Thank You

Osama Mustafa

Remove Node From Oracle RAC 10g (Part-1)

Posted on April 17, 2012December 23, 2018 by Osama Mustafa in Database

In This Article I Will Show You How to Remove Node From Oracle Real Application Cluster 10g :

Let’s Start :

Main Steps To Remove Node From Oracle Real Application Cluster :

1.Delete the instance on the node to be deleted.

2.Clean up the ASM instance.

3.Remove the listener from the node to be deleted.

4.Remove the node from the database.

5.Remove the node from ASM.

6.Remove ONS configuration from the node to be deleted.

7.Remove the node from the cluster-ware.

**The First Steps Is To Delete Instance On The Node (From The Node you want to delete) —- > DBCA

On The Above Message Press Ok .

**Clean Up The ASM Instance : (From node you want to delete)

you need to manually remove the initialization parameter file of that ASM instance

remove all the log files of that ASM instance. These files are generally located in the $ORACLE_BASE/admin directory then remove the associated ASM entry from the /etc/oratabfile

Remove The Listener (From the Node You want to delete ): ——> NETCA

Note : You need to choose the Node you want to be deleted , Not Select All .

Part Two In Separate Article

Oracle Rac 10g Add Node Steps

Posted on April 13, 2012December 23, 2018 by Osama Mustafa in Database

This Article Discuss Adding/Remove Node In Oracle Real application Cluster 10g The below Steps easy all you have to do is follow up with them Let’s Start :

ADD NODE :

1-Check Pre-requisities On All Node , This Include Old Node And New Node .

cluvfy stage -pre crsinst -n -r 10gR2

2-From the First Node , Execute the following Command :

/oui/bin/addNode.sh

The Final Result For Step 2 .

3-We need to configure new ONS (From The First Node) :

cd cluster_home/bin
cat cluster_home/opmn/conf/ons.config (to get the remoteport)
./racgons add_config Node-name:6200 (remoteport)

4-Optional : needed Only If Specific Home Directory to host ASM (From the First Node)

cluster_home/bin/addnode.sh

5- From The First node we need to Add RAC

cd cluster_home/bin

./addnode.sh

6-Add Listener , Do This Steps From The New Node On Terminal

netca –> Cluster Configuration —>Name of New Node

7-Final Step , Add Database Instance To new node ( from the First Node ) Using

DBCA —> Oracle Real Application Cluster —> Instance Management —> Add Instance

You Need Follow Up with Screen as Normal Installation Of Database

–List of cluster databases.select your RAC database and enter SYS credentials

– List of cluster database instances , next

–Instance naming and node selection . Instance Name

Note :

if you are using ASM for your database storage,the DBCA detects the need for an ASM instance creation on the new nodeThis must be done before the DBCA can create the database instance on that node. Click Yes.

Thank You
Osama Mustafa

I will Talk Later How To Remove Node From Oracle Real Application Cluster .

Change VIP Addresses

Posted on April 11, 2012December 23, 2018 by Osama Mustafa in Database

1.Determine the interface used to support your VIP:

$ ifconfig -a

2.Stop all resources depending on the VIP:

$ srvctl stop instance -d DB -i DB1

$ srvctl stop asm -n node1

# srvctl stop nodeapps -n node1

3.Verify that the VIP is no longer running:

$ ifconfig -a

$ crs_stat

4.Change IP in /etc/hostsand DNS.

5.Modify your VIP address using srvctl:

# srvctl modify nodeapps -n node1 -A 192.168.2.125/255.255.255.0/eth0

6.Start nodeappsand all resources depending on it:

# srvctl start nodeapps -n node1

7.Repeat from step 1 for the next node.

Thank You

Osama Mustafa

Some Basic To backup Voting Disk And OCR For Oracle RAC

Posted on April 11, 2012December 23, 2018 by Osama Mustafa in Database

We Need To Backup Voting Disk and OCR For Oracle RAC For Disaster Situation :

Add Voting Disk :

# crsctl add css votedisk

To remove a voting disk:

# crsctl delete css votedisk

If Oracle Clusterware is down on all nodes, use the –force option:

# crsctl add css votedisk -force
# crsctl delete css votedisk -force

Backup Voting Disk :

$ crsctl query css votedisk
$ dd if= of= bs=4k

OCR BACKUP :

Oracle RAC Taking Backup For OCR By Default , You will Find It On :

cd $ORACLE_BASE/Crs/cdata/jfv_clus

Change the default automatic backup location:

# ocrconfig –backuploc /shared/bak

Back Up OCR Manually :

# ocrconfig –export file name

Recover OCR Using Physical Backups:

1.Locate a physical backup:

$ ocrconfig –showbackup

2.Review its contents:

# ocrdump –backupfile file_name

3.Stop Oracle Clusterwareon all nodes:

# crsctl stop crs

4.Restore the physical OCR backup:

# ocrconfig –restore <CRS HOME>/cdata/jfv_clus/day.ocr

5.Restart Oracle Clusterware on all nodes:

# crsctl start crs

6.Check OCR integrity:

$ cluvfy comp ocr -n all

Thank You
Osama Mustafa

UNKNOWN State Of RAC Resources

Posted on April 10, 2012December 23, 2018 by Osama Mustafa in Database

Sometimes when you check Oracle Cluster Status Via “crs_stat -t” :

------------------------------------------------------------
ora.orcl.db    application    ONLINE    ONLINE    rac1
ora....11.inst application    ONLINE    ONLINE    rac1
ora....SM1.asm application    ONLINE    ONLINE    rac1
ora....DC.lsnr application    ONLINE    ONLINE    rac1
ora....idc.gsd application    ONLINE    ONLINE    rac1
ora....idc.ons application    ONLINE    ONLINE    rac1
ora....idc.vip application    ONLINE    ONLINE    rac1
ora....SM2.asm application    ONLINE    UNKNOWN   rac2
ora....C2.lsnr application    ONLINE    ONLINE    rac2
ora....dc2.gsd application    ONLINE    ONLINE    rac2
ora....dc2.ons application    ONLINE    ONLINE    rac2
ora....dc2.vip application    ONLINE    ONLINE    rac2

If You try to restart Oracle Cluster via

./crs_stop -all
./crs_start -all

It will not working so you have to do the following simple steps :

First :

crs_stop -f Service_name

Example :

crs_stop -f ora.rac2.ASM2.asm

Second :

crs_start -f services_name

Example :
crs_start -f ora.rac2.ASM2.asm 

Thank you
Osama

Create Enterprise manager for RAC (Real Application Cluster)

Posted on November 10, 2011December 23, 2018 by Osama Mustafa in Database

Hi All ,

Sometimes we need to create enterprise manager for Rac Environment , its the same way for One instance with different Command .

We are here talk about 10g , 11g database .
you follow the below steps to create or recreate enterprise manager LET’S START :

if you already have RAC – enterprise manager and you need to recreate then start from here

**Stop Dbconsole for the both nodes :

Commands :

On Node-1
emctl stop dbconsole .
On Node 2 :
emctl stop dbconsole .

**Drop dbconsole with re posterity from node 1 :

Command :

$emca -deconfig dbcontrol db -repos drop -cluster

“you will wait for while until repos been dropped”.

**recreate dbconsole with re posterity from node 1 “if you don’t have EM for your Rac start from here”

Command :

$emca -config dbcontrol db -repos create -cluster

“follow the instsruction and fill the information such as DB SID,Listener Port , Crs Name”

To know your CRS NAME :

$CRS_HOME/bin/cemutlo -n

the output will be your crs name .

Thank you
Osama mustafa