Tag: cluster

ORA-29701: Unable to connect to Cluster Manager

Posted on by Osama Mustafa in Database
ORA-29701: Unable to connect to Cluster Manager.

The reason is Oracle cssd daemon process was not running.

As Root User :

Step 1: Go to the /etc folder
Step 2: Execute the following command.

# init.cssd start

startup will be queued to init within 30 seconds

Step 3: Wait for 2 minutes and execute the following commands

# cd $ORACLE_HOME/bin
# ./localconfig delete

Stopping CSSD
Failure in CSS initialization opening OCR.
Shutdown has begun. The daemons should exit soon.

Step 4: Execute the following command

# ./localconfig add

Successfully accumulated necessary OCR keys.
Creating OCR keys for user root, privgrp ‘system’
Operation Successful
Configuration for local css has been initialized.
Adding to inittab
Startup will be queued to init within 30 seconds
Checking the status of new oracle init process..
Expecting the CRS daemons to be up within 600 seconds
CSS is active on these nodes
nodelist
CSS is active on all nodes
Oracle CSS service is installed and running under init(1M)

Step 5: Start the ASM instance now.

Oracle Real Application Cluster Managment

Posted on by Osama Mustafa in Database
1. CRSCTL: 

Cluster Control utility performs various administrative operation of oracle clusterware. It is located in $ORA_CRS_HOME/bin and must be executed by the “root” user.

a. To check the current state of all oracle clusterware daemon:

[root@rac1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

b.You can also check the state of individual oracle clusterware daemon:

[root@rac1 bin]# ./crsctl check cssd
CSS appears healthy

[root@rac1 bin]# ./crsctl check crsd
CRS appears healthy

[root@rac1 bin]# ./crsctl check evmd
EVM appears healthy

c. To start oracle clusterware

[root@rac1 bin]# ./crsctl start crs

Attempting to start CRS stack
The CRS stack will be started shortly

d. To stop oracle clusterware

[root@rac1 bin]# ./crsctl stop crs
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.

e. To disable  oracle clusterware:

[root@rac1 bin]# ./crsctl disable crs

f.  To enable  oracle clusterware:

[root@rac1 bin]# ./crsctl enable crs

g. To get current value of CSS parameter

[root@rac1 bin]# ./crsctl get css
for  example: to get value of misscount parameter
 [root@rac1 bin]# ./crsctl get css misscount
60

h. To set a new value of CSS parameter

[root@rac1 bin]# ./crsctl set css
for  example: to set value of misscount parameter

[root@rac1 bin]# ./crsctl set css misscount 120
Configuration parameter misscount is now set to 120.

i. To unset CSS parameter value

[root@rac1 bin]# ./crsctl unset css
for  example: to unset value of misscount parameter

[root@rac1 bin]# ./crsctl unset css misscount
Configuration parameter misscount is now undefined.

j. To list the module for debugging in CSS

[root@rac2 bin]# ./crsctl lsmodules css

The following are the CSS modules ::
CSSD
COMMCRS
COMMNS

2. CRS_STAT:
 It reports the current state of resources configured in the OCR.

[oracle@rac1 bin]$ ./crs_stat -t
Name                      Type              Target       State         Host
———————————————————————————–
ora….C1.inst             application    ONLINE    ONLINE          rac1
ora….C2.inst             application    ONLINE    ONLINE          rac2
ora….AC1.srv            application    ONLINE    ONLINE          rac1
ora.RAC.abc.cs          application    ONLINE    ONLINE          rac1
ora.RAC.db                application    ONLINE    ONLINE         rac2
ora….AC1.srv            application    ONLINE    ONLINE          rac1
ora….ice2.cs             application    ONLINE    ONLINE          rac1
ora….AC1.srv            application    ONLINE    ONLINE          rac1

3. CRS_STOP:

 This command used to stop resource or cluster member.

[root@rac1 bin]# ./crs_stop ora.rac1.ons
Attempting to stop `ora.rac1.ons` on member `rac1`
Stop of `ora.rac1.ons` on member `rac1` succeeded.

4.  CRS_START:

 This command used to start resource or cluster member.

[root@rac1 bin]# ./crs_start ora.rac1.ons
Attempting to start `ora.rac1.ons` on member `rac1`
Start of `ora.rac1.ons` on member `rac1` succeeded.

5.  OCRDUMP :

 It dumps the contents of OCR into a text file.

 [root@rac1 bin]# ./ocrdump /home/oracle/ocr.dmp

6. OCRCHECK :

 It verifies the integrity of the OCR.

[root@rac2 bin]# ./ocrcheck

Status of Oracle Cluster Registry is as follows :
Version                                :    2
Total space (kbytes)         :    5237072
Used space (kbytes)         :    9360
Available space (kbytes)  :    5227712
ID                                        :    794527192
Device/File Name             :    /apps/oracle/oradata/ocr
Device/File integrity check succeeded
Cluster registry integrity check succeeded

7. OCRCONFIG:

 It perform various administrative operation on the OCR.

CRS-4640 Error on Starting 11gR2 clusterware

Posted on by Osama Mustafa in Database 
ERROR :  
 
[root@RAC01 cssd]# /oragrid/product/11.2/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
 
When Trying To start Oracle Cluster with following Command :
 
[root@RAC01 cssd]# /oragrid/product/11.2/bin/crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors. 

When Trying To Stop Oracle Cluster With Following Command : 

crsctl stop crs command failed

[root@RAC01 cssd]# /oragrid/product/11.2/bin/crsctl stop crs
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.


Solution is Simple :
 
Just Run 
 
[root@RAC01 cssd]# /oragrid/product/11.2/bin/crsctl start cluster 

[root@RAC01 ~]# /oragrid/product/11.2/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

Thank you
Osama mustafa

Change to Archive Log Mode In Oracle Rac

Posted on by Osama Mustafa in Database
1. Disable clustering putting cluster_database parameter FALSE.

$export ORACLE_SID=ORCL1
$sqlplus “/ as sysdba”

Check the status of archive mode of the database:

SQL>archive log list

Database log mode              No Archive Mode
Automatic archival             Disabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     407
Current log sequence           408

SQL> alter system set cluster_database=false scope=spfile sid=’ORCL1′;

2. Shutdown all instances using srvctl utilty

$ srvctl stop database -d cobra

3. Mount tha database using one of the instance:

$ sqlplus “/ as sysdba”

SQL> startup mount

4. Enable archivelog using following command:

SQL> alter database archivelog;

5. Re-enable clustering putting instance parameter cluster_database to TRUE from the current instance:

SQL> alter system set cluster_database=true scope=spfile sid=’ORCL1′;

6. Shutdown the local instance:

SQL> shutdown immediate

7. Startup all instances using srvctl utility:

#srvctl start database -d orcl

8. If any service is not up then get up those using srvctl utility:

#srvctl start service -dorcl

9. And now check archivelog mode is enabled or not using following:

$sqlplus “/ as sysdba”

SQL> archive log list

Database log mode            Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     407
Current log sequence           408

Thank you
Osama mustafa

DBCONSOLE For Real Application Cluster

Posted on by Osama Mustafa in Database
Drop DBCONSOLE FOR RAC (Uninstall) : 



we will user the following command :

emca -deconfig dbcontrol db -repos drop -cluster

Make sure you have the below information :

  • Database unique name
  • Listener port number
  • Password for SYS user
  • Password for SYSMAN user

Create DBCONSOLE FOR RAC ( Install) : 

In this case you have more than one choice you can use what you want depend on your requirements :

1-

emca -config dbcontrol db -repos create -cluster

2-

 emca -reconfig dbcontrol –cluster –EM_NODE node1 -EM_NODE_LIST node2,node3,node4

emca -reconfig dbcontrol –cluster –EM_NODE node5 -EM_NODE_LIST node6,node7,node8

 The below Information for installing DBCONSOLE :

  • The Database unique name
  • The Listener port number
  • The Cluster name
  • The Password for SYS user
  • The Password for DBSNMP user
  • The Password for SYSMAN user
  • The Email address for notifications (optional)
  • The Outgoing Mail (SMTP) server for notifications (optional)
  • The ASM ORACLE_HOME e.g. /u01/app/asm/product/10.2.0
  • The ASM port e.g. 1521
  • The ASM user role e.g. SYSDBA
  • The ASM username e.g. SYS
  • The ASM user password

Note : Sometimes you need to put the password between “” . 
Thank you
Osama mustafa

Oracle Rac 10g Add Node Steps

Posted on by Osama Mustafa in Database
This Article Discuss Adding/Remove Node In Oracle Real application Cluster 10g The below Steps easy all you have to do is follow up with them Let’s Start :

ADD NODE : 

1-Check Pre-requisities On All Node , This Include Old Node And New Node .

cluvfy  stage -pre crsinst -n -r 10gR2

 2-From the First Node , Execute the following Command :

/oui/bin/addNode.sh

The Final Result For Step 2 .

3-We need to configure new ONS (From The First Node) :

cd cluster_home/bin
cat cluster_home/opmn/conf/ons.config (to get the remoteport)
./racgons add_config Node-name:6200 (remoteport)

4-Optional : needed Only If Specific Home Directory to host ASM (From the First Node)

cluster_home/bin/addnode.sh

5- From The First node we need to Add RAC

cd cluster_home/bin

./addnode.sh

6-Add Listener , Do This Steps From The New Node On Terminal

netca –> Cluster Configuration —>Name of New Node

7-Final Step , Add Database Instance To new node ( from the First Node ) Using

DBCA —> Oracle Real Application Cluster —> Instance Management —> Add Instance

 

\

You Need Follow Up with Screen as Normal Installation Of Database
–List of cluster databases.select your RAC database and enter SYS credentials 
– List of cluster database instances , next
–Instance naming and node selection . Instance Name
Note : 
 
if you are using ASM for your database storage,the DBCA detects the need for an ASM instance creation on the new nodeThis must be done before the DBCA can create the database instance on that node. Click Yes.

Thank You
Osama Mustafa

I will Talk Later How To Remove Node From Oracle Real Application Cluster .

Some Basic To backup Voting Disk And OCR For Oracle RAC

Posted on by Osama Mustafa in Database
We Need To Backup Voting Disk and OCR For Oracle RAC For Disaster Situation :

Add Voting Disk :

# crsctl add css votedisk

To remove a voting disk:

# crsctl delete css votedisk

If Oracle Clusterware is down on all nodes, use the –force option:

# crsctl add css votedisk -force
# crsctl delete css votedisk -force

 Backup Voting Disk :

$ crsctl query css votedisk
$ dd if= of= bs=4k

OCR BACKUP :

Oracle RAC Taking Backup For OCR By Default , You will Find It On :

 cd $ORACLE_BASE/Crs/cdata/jfv_clus

Change the default automatic backup location:

# ocrconfig –backuploc /shared/bak

Back Up OCR Manually :

 # ocrconfig –export file name

 Recover OCR Using Physical Backups:

1.Locate a physical backup:
 $ ocrconfig –showbackup
2.Review its contents:

# ocrdump –backupfile file_name
 3.Stop Oracle Clusterwareon all nodes:

 # crsctl stop crs
4.Restore the physical OCR backup:

# ocrconfig –restore <CRS HOME>/cdata/jfv_clus/day.ocr

5.Restart Oracle Clusterware on all nodes:

 # crsctl start crs
6.Check OCR integrity:

$ cluvfy comp ocr -n all

Thank You
Osama Mustafa 

 



UNKNOWN State Of RAC Resources

Posted on by Osama Mustafa in Database
Sometimes when you check Oracle Cluster Status Via “crs_stat -t” :

------------------------------------------------------------
ora.orcl.db    application    ONLINE    ONLINE    rac1
ora....11.inst application    ONLINE    ONLINE    rac1
ora....SM1.asm application    ONLINE    ONLINE    rac1
ora....DC.lsnr application    ONLINE    ONLINE    rac1
ora....idc.gsd application    ONLINE    ONLINE    rac1
ora....idc.ons application    ONLINE    ONLINE    rac1
ora....idc.vip application    ONLINE    ONLINE    rac1
ora....SM2.asm application    ONLINE    UNKNOWN   rac2
ora....C2.lsnr application    ONLINE    ONLINE    rac2
ora....dc2.gsd application    ONLINE    ONLINE    rac2
ora....dc2.ons application    ONLINE    ONLINE    rac2
ora....dc2.vip application    ONLINE    ONLINE    rac2

If You try to restart Oracle Cluster via

./crs_stop -all
./crs_start -all

It will not working so you have to do the following simple steps :

 First :

crs_stop -f Service_name
 
Example :
crs_stop -f ora.rac2.ASM2.asm

 Second :

crs_start -f services_name

Example :

crs_start -f ora.rac2.ASM2.asm

Thank you
Osama

ocfs2console: o2cb_ctl: Unable to access cluster service while creating node

Posted on by Osama Mustafa in Database
For the first time, during configuring the nodes, ocfs2console fails with message:
“o2cb_ctl: Unable to access cluster service while creating node”
TODO:
– Quit ocfs2console
– Stop the service
– Remove file /etc/ocfs2/cluster.conf
– Restart ocfs2console
– Configure the nodes again

[root@rac1 ~]#  /etc/init.d/ocfs2 stop
Stopping Oracle Cluster File System (OCFS2)                [  OK  ]

[root@rac1 ~]# /etc/init.d/o2cb offline ocfs2

[root@rac1 ~]# /etc/init.d/o2cb unload
Unmounting ocfs2_dlmfs filesystem: OK
Unloading module "ocfs2_dlmfs": OK
Unmounting configfs filesystem: OK
Unloading module "configfs": OK

[root@rac1 ~]# rm -f /etc/ocfs2/cluster.conf
[root@rac1 ~]# /usr/sbin/ocfs2console &