Month: April 2012

Remove Node From Oracle RAC 10g (Part-1)

Posted on by Osama Mustafa in Database
In This Article I Will Show You How to Remove Node From Oracle Real Application Cluster 10g :

Let’s Start : 

Main Steps To Remove Node From Oracle Real Application Cluster : 

1.Delete the instance on the node to be deleted.
2.Clean up the ASM instance.
3.Remove the listener from the node to be deleted.
4.Remove the node from the database.
5.Remove the node from ASM.
6.Remove ONS configuration from the node to be deleted.
7.Remove the node from the cluster-ware.
 **The First Steps Is To Delete Instance On The Node (From The Node you want to delete) —- >  DBCA
 
  
On The Above Message Press Ok .
**Clean Up The ASM Instance : (From node you want to delete) 

 
 
you need to manually remove the initialization parameter file of that ASM instance
 
remove all the log files of that ASM instance. These files are generally located in the $ORACLE_BASE/admin directory then remove the associated ASM entry from the /etc/oratabfile
 
Remove The Listener (From the Node You want to delete ):  ——> NETCA

Note : You need to choose the Node you want to be deleted , Not Select All .

Part Two In Separate Article 

Oracle Rac 10g Add Node Steps

Posted on by Osama Mustafa in Database
This Article Discuss Adding/Remove Node In Oracle Real application Cluster 10g The below Steps easy all you have to do is follow up with them Let’s Start :

ADD NODE : 

1-Check Pre-requisities On All Node , This Include Old Node And New Node .

cluvfy  stage -pre crsinst -n -r 10gR2

 2-From the First Node , Execute the following Command :

/oui/bin/addNode.sh

The Final Result For Step 2 .

3-We need to configure new ONS (From The First Node) :

cd cluster_home/bin
cat cluster_home/opmn/conf/ons.config (to get the remoteport)
./racgons add_config Node-name:6200 (remoteport)

4-Optional : needed Only If Specific Home Directory to host ASM (From the First Node)

cluster_home/bin/addnode.sh

5- From The First node we need to Add RAC

cd cluster_home/bin

./addnode.sh

6-Add Listener , Do This Steps From The New Node On Terminal

netca –> Cluster Configuration —>Name of New Node

7-Final Step , Add Database Instance To new node ( from the First Node ) Using

DBCA —> Oracle Real Application Cluster —> Instance Management —> Add Instance

 

\

You Need Follow Up with Screen as Normal Installation Of Database
–List of cluster databases.select your RAC database and enter SYS credentials 
– List of cluster database instances , next
–Instance naming and node selection . Instance Name
Note : 
 
if you are using ASM for your database storage,the DBCA detects the need for an ASM instance creation on the new nodeThis must be done before the DBCA can create the database instance on that node. Click Yes.

Thank You
Osama Mustafa

I will Talk Later How To Remove Node From Oracle Real Application Cluster .

Dealing With Database Gaurd

Posted on by Osama Mustafa in Database

Database Switchover 

Using this method you can switch backwards and forwards between the primary and DR servers (e.g. so that the primary can become DR and DR can become primary) without having to rebuild either environment:

 

On Primary Server:
SQL> alter database commit to switchover to standby;
This may cause the following error to be generated:
ERROR at line 1:
ORA-01093: ALTER DATABASE CLOSE only permitted with no sessions connected
If this does occur then restart the database, as below, before retrying the above command:
SQL> shutdown immediate
SQL> startup

 

SQL> shutdown immediate
SQL> startup nomount
SQL> alter database mount standby database;
SQL> alter database recover managed standby database disconnect;

The primary server is now configured as a DR standby database.
On DR Server:
SQL> alter database recover managed standby database cancel;
SQL> alter database commit to switchover to primary;
SQL> shutdown immediate
SQL> startup
The DR server is now configured as the primary database.
To switch back you just need to repeat the above process but the other way around (e.g. convert the DR database back to a standby and the primary database back to primary).

Activating a Standby Database

If the primary database is not available the standby database can be converted into the primary database as follows:


SQL> alter database recover managed standby database cancel;
SQL> alter database activate standby database;
SQL> shutdown immediate
SQL> startup
The original primary database is now obsolete and can be rebuilt as a standby database once it is available again.

Opening the Standby Database in Read Only Mode

 

The standby database can be opened in read only mode for querying and then converted back into a standby database without affecting the primary.
On standby server:
SQL> alter database recover managed standby database cancel;
SQL> alter database open read only;
The standby database is now open and available for querying in read only mode.
To put the standby database back into standby mode:
SQL> shutdown immediate
SQL> startup nomount
SQL> alter database mount standby database;
SQL> alter database recover managed standby database disconnect;

How to check whether the Standby Database is in Sync

On the primary server:

SQL> SELECT max(sequence#) AS “PRIMARY” FROM v$log_history;

 
On the standby server:

SQL> SELECT max(sequence#) AS “STANDBY”, applied
          FROM v$archived_log GROUP BY applied;

The standby database is in sync with the primary database if the above PRIMARY value matches the above STANDBY value where applied = ‘YES’.
 

 


Change VIP Addresses

Posted on by Osama Mustafa in Database
1.Determine the interface used to support your VIP:
 $ ifconfig -a
2.Stop all resources depending on the VIP:
$ srvctl stop instance -d DB -i DB1
$ srvctl stop asm -n node1
# srvctl stop nodeapps -n node1

3.Verify that the VIP is no longer running:

$ ifconfig -a
$ crs_stat
4.Change IP in /etc/hostsand DNS.

5.Modify your VIP address using srvctl:
# srvctl modify nodeapps -n node1 -A 192.168.2.125/255.255.255.0/eth0
6.Start nodeappsand all resources depending on it:
# srvctl start nodeapps -n node1
7.Repeat from step 1 for the next node.
Thank You 
Osama Mustafa 

 

 

 

Some Basic To backup Voting Disk And OCR For Oracle RAC

Posted on by Osama Mustafa in Database
We Need To Backup Voting Disk and OCR For Oracle RAC For Disaster Situation :

Add Voting Disk :

# crsctl add css votedisk

To remove a voting disk:

# crsctl delete css votedisk

If Oracle Clusterware is down on all nodes, use the –force option:

# crsctl add css votedisk -force
# crsctl delete css votedisk -force

 Backup Voting Disk :

$ crsctl query css votedisk
$ dd if= of= bs=4k

OCR BACKUP :

Oracle RAC Taking Backup For OCR By Default , You will Find It On :

 cd $ORACLE_BASE/Crs/cdata/jfv_clus

Change the default automatic backup location:

# ocrconfig –backuploc /shared/bak

Back Up OCR Manually :

 # ocrconfig –export file name

 Recover OCR Using Physical Backups:

1.Locate a physical backup:
 $ ocrconfig –showbackup
2.Review its contents:

# ocrdump –backupfile file_name
 3.Stop Oracle Clusterwareon all nodes:

 # crsctl stop crs
4.Restore the physical OCR backup:

# ocrconfig –restore <CRS HOME>/cdata/jfv_clus/day.ocr

5.Restart Oracle Clusterware on all nodes:

 # crsctl start crs
6.Check OCR integrity:

$ cluvfy comp ocr -n all

Thank You
Osama Mustafa 

 



ORA-16139: media recovery required

Posted on by Osama Mustafa in Database
During switch physical standby database to primary database i received following error message on (physical standby database)

ٍٍSQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY;

ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY

*ERROR at line 1:ORA-16139: media recovery required

THE SOLUTION :

SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE;

SWITCHOVER_STATUS

——————–

SWITCHOVER LATENT

SQL> RECOVER MANAGED STANDBY DATABASE FINISH;

Media recovery complete.

SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY;

 Metalink Note:[ID 219554.1]

Thank You
osama

UNKNOWN State Of RAC Resources

Posted on by Osama Mustafa in Database
Sometimes when you check Oracle Cluster Status Via “crs_stat -t” :

------------------------------------------------------------
ora.orcl.db    application    ONLINE    ONLINE    rac1
ora....11.inst application    ONLINE    ONLINE    rac1
ora....SM1.asm application    ONLINE    ONLINE    rac1
ora....DC.lsnr application    ONLINE    ONLINE    rac1
ora....idc.gsd application    ONLINE    ONLINE    rac1
ora....idc.ons application    ONLINE    ONLINE    rac1
ora....idc.vip application    ONLINE    ONLINE    rac1
ora....SM2.asm application    ONLINE    UNKNOWN   rac2
ora....C2.lsnr application    ONLINE    ONLINE    rac2
ora....dc2.gsd application    ONLINE    ONLINE    rac2
ora....dc2.ons application    ONLINE    ONLINE    rac2
ora....dc2.vip application    ONLINE    ONLINE    rac2

If You try to restart Oracle Cluster via

./crs_stop -all
./crs_start -all

It will not working so you have to do the following simple steps :

 First :

crs_stop -f Service_name
 
Example :
crs_stop -f ora.rac2.ASM2.asm

 Second :

crs_start -f services_name

Example :

crs_start -f ora.rac2.ASM2.asm

Thank you
Osama

Checking O2CB heartbeat: Not active

Posted on by Osama Mustafa in Database

[root@rac01]# /etc/init.d/o2cb  status

Module “configfs”: Loaded
Filesystem “configfs”: Mounted
Module “ocfs2_nodemanager”: Loaded
Module “ocfs2_dlm”: Loaded
Module “ocfs2_dlmfs”: Loaded
Filesystem “ocfs2_dlmfs”: Mounted
Checking O2CB cluster ocfs2: Online
  Heartbeat dead threshold: 61
  Network idle timeout: 30000
  Network keepalive delay: 2000
  Network reconnect delay: 2000
Checking O2CB heartbeat: Not active

The Solution like the Following :

[root@rac01]# /etc/init.d/o2cb  configure

Configuring the O2CB driver.
This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot.  The current values will be shown in brackets (‘[]’).  Hitting
without typing an answer will keep that current value.  Ctrl-C
will abort.
Load O2CB driver on boot (y/n) [y]: y
Cluster to start on boot (Enter “none” to clear) [ocfs2]:
Specify heartbeat dead threshold (>=7) [31]: 61
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
O2CB cluster ocfs2 already online

[root@rac01]# /etc/init.d/o2cb  stop
Stopping O2CB cluster ocfs2: OK
Unmounting ocfs2_dlmfs filesystem: OK
Unloading module “ocfs2_dlmfs”: OK
Unmounting configfs filesystem: OK
Unloading module “configfs”: OK

 [root@rac01]# /etc/init.d/o2cb  start
Loading module “configfs”: OK
Mounting configfs filesystem at /config: OK
Loading module “ocfs2_nodemanager”: OK
Loading module “ocfs2_dlm”: OK
Loading module “ocfs2_dlmfs”: OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK

Now

  [root@rac01]# ocfs2console

Then Mount Command .

How To Check Oracle Physical Standby is in Sync with the Primary or Not?

Posted on by Osama Mustafa in Database
On Primary

set pages 1000
set lines 120
column DEST_NAME format a20
column DESTINATION format a35
column ARCHIVER format a10
column TARGET format a15
column status format a10
column error format a15
select DEST_ID,DEST_NAME,DESTINATION,TARGET,STATUS,ERROR from v$archive_dest
where DESTINATION is NOT NULL

At Physical Standby


SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference"
FROM
(SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH,
(SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL
WHERE
ARCH.THREAD# = APPL.THREAD#
ORDER BY 1



SELECT THREAD#, LOW_SEQUENCE#, HIGH_SEQUENCE# FROM V$ARCHIVE_GAP



select process,status,sequence# from v$managed_standby

Compare the output of the query its should be equal .

Instance Recovery and RAC

Posted on by Osama Mustafa in Database

Here are some guidelines you can use to make sure that instance recovery in your RAC environment is faster:
•Use parallel instance recovery by setting RECOVERY_PARALLISM.
•Increase PARALLEL_EXECUTION_MESSAGE_SIZE from its default of 2,148 bytes to 4 KB or 8 KB. This should provide better recovery slave performance.
•Set PARALLEL_MIN_SERVERS to CPU_COUNT-1. This will prespawnrecovery slaves at startup time.
•Using asynchronous I/O is one of the most crucial factors in recovery time. The first-pass log read uses asynchronous I/O.
•Instance recovery uses 50 percent of the default buffer cache for recovery buffers. If this is not enough, some of the steps of instance recovery will be done in several passes. You should be able to identify such situations by looking at your alert.logfile. In that case, you should increase the size of your default buffer cache.