I uploaded the document on Slideshare , check it from here
Cheers
Osama Mustafa
For the people who think differently Welcome aboard
I uploaded the document on Slideshare , check it from here
Cheers
Osama Mustafa
./sshUserSetup.sh
useful script to establish and exchange ssh keys between all the nodes of the cluster called sshUserSetup.sh
And Can Be Used like the below :-
./sshUserSetup.sh -user oracle -hosts “node1 node2” -noPromptPassphrase
./sshUserSetup.sh -user grid -hosts “node1 node2” -noPromptPassphrase
./runcluvfy.sh
Cluster Verification Utility
Regarding to Oracle documentation :-
“performs system checks in preparation for installation, patch updates, or other system changes. Using CVU ensures that you have completed the required system configuration and preinstallation steps so that your Oracle Grid Infrastructure or Oracle Real Application Clusters (Oracle RAC) installation, update, or patch operation completes successfully.”
you can learn More Here
Example :-
./runcluvfy.sh stage -pre crsinst -n node1,node -fixup -verbose
The Error in the logs like the below :-
CRS-5011:Check of resource “+ASM” failed: details at “(:CLSN00006:)”
Node #2 was running without any problem , the issue was with node #1 and nothing changed recently.
The Error in Node #1 :-
crsctl start cluster
PRCR-1079 : Failed to start resource ora.asm
ORA-01031: insufficient privileges
CRS-5017: The resource action “ora.asm start” encountered the following error:
ORA-01031: insufficient privileges
. For details refer to “(:CLSN00107:)” in “/u01/app/grid/11.2.0.4/log/lux335/agent/ohasd/oraagent_oracle/oraagent_oracle.log”.
as you see from the above error “ORA-01031: insufficient privileges” i was running this command as root.
After check node #1 the solution so simple :-
Under $GRID_HOME/network/admin
vi sqlnet.ora
SQLNET.AUTHENTICATION_SERVICES = (NONE)
which is wrong and to be changed to the following value :-
SQLNET.AUTHENTICATION_SERVICES = (NTS)
Reboot Node #1 and everything running successfully.
Thank you
Osama mustafa
Database 11gr2
Rac 11gr2
When trying to expdp the below error appear :
expdp dumpfile=tdmp_test.dmp logfile=1.log directory=expdp_folder schemas=siebel parallel=4
ORA-31693: Table data object “CS_XM_TEMP”.”CS_XM_TEMP_CC” failed to load/unload and is being skipped due to error:
ORA-31617: unable to open dump file “/u01/expdp /dump_test.dmp” for write
ORA-19505: failed to identify file “/u01/expdp/dump_test.dmp”
ORA-27037: unable to obtain file status
You need to make sure of the below :
– Folder exdp_folder should be exists on the both nodes with same path.
even if you try to connect using tnsnames it will generate error so solution
-Remove parallel from your expdp command to be like this
expdp dumpfile=tdmp_test.dmp logfile=1.log directory=expdp_folder schemas=siebel
-Use Cluster Option in expdp
expdp dumpfile=tdmp_test.dmp logfile=1.log directory=expdp_folder schemas=siebel cluster=n
Thank you
Osama mustafa
Voting Disk: It manage information about node membership. Each voting disk must be accessible by all nodes in the cluster.If any node is not passing heat-beat across other note or voting disk, then that node will be evicted by Voting disk.
Minimum 1 and maximum 15 copy of voting disk is possible.
[oracle@rsingle ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
– —– —————– ——— ———
1. ONLINE 6a60a2c3510c4fbfbff62dcdc279b247 (ORCL:DATA1) [DATA]
[root@rsingle ~]# /u01/app/11.2.0/grid/bin/crsctl replace votedisk +CRS
Successful addition of voting disk afb77b2693a24f1ebfe876784103e82a.
Successful addition of voting disk 3e2542c5b1154ffdbfc8b6dea7dce390.
Successful addition of voting disk 8e0f3c5921cc4f93bf223de1465d83cc.
Successful deletion of voting disk 6a60a2c3510c4fbfbff62dcdc279b247.
Successfully replaced voting disk group with +CRS.
CRS-4266: Voting file(s) successfully replaced
[root@rsingle ~]# /u01/app/11.2.0/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
– —– —————– ——— ———
1. ONLINE afb77b2693a24f1ebfe876784103e82a (ORCL:DATA2) [CRS]
2. ONLINE 3e2542c5b1154ffdbfc8b6dea7dce390 (ORCL:DATA3) [CRS]
3. ONLINE 8e0f3c5921cc4f93bf223de1465d83cc (ORCL:DATA4) [CRS]
Located 3 voting disk(s).
[oracle@rsingle ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2964
Available space (kbytes) : 259156
ID : 1390115973
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Create mirror copy of OCR online
[oracle@rsingle ~]$ sudo ocrconfig -add +CRS
Password:
[root@rsingle ~]# /u01/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2964
Available space (kbytes) : 259156
ID : 1390115973
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File Name : +CRS
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@rsingle ~]# cat /etc/oracle/ocr.loc
#Device/file getting replaced by device +CRS
ocrconfig_loc=+DATA
ocrmirrorconfig_loc=+CRS
Share it for knowledge
Thank you atual
Osama mustafa
There could be lot of applications to using this feature of callouts. For example, when an instance goes down, we all know that services running on that instance are relocated to other available instances. But when that instance comes back up, those relocated services need to be manually put back to their preferred instance. By using FAN callouts, we can automate this task.
1. Go to $ORA_CRS_HOME/racg and create usrco directory on all the nodes. So the server side callout directory would look something like this:
/oracle/product/crs/racg/usrco
2. Place your callout scripts under this dir. This will be called on every state change. You could use any executable like shell script or a perl script.
I have tested this on a 2-node Oracle RAC database version 10.2.0.2 on x64 RHEL4U7.
Save the below sample script as /oracle/product/crs/racg/usrco/instup_relocate.pl
This one traps the INSTANCE UP event and calls another script to relocate the services
#!/usr/local/bin/perl
# instup_relocate.pl
# This is a callout program that will, on an INSTANCE UP event relocate services back
# This script is supposed to reside in $CRS_HOME/racg/usrco as an executable on all the nodes. “usrco” directory needs to be created for callouts.
use strict;
# Define Oracle and Crs Home
my $CRS_HOME=”/oracle/product/crs”;
my $ORACLE_HOME=”/oracle/product/10.2″;
# TMP refers to the log location only
my $TMP = “/tmp”;
# Enable logging
my $LOGFILE = “$TMP/SRV_co.log”;
# Define variables that would be captured by callout event
my $instance;
my $database;
my $host;
my $service;
my $reason;
my $card;
my $status;
my ($key,$value) = “”;
# Open logfile
local *LOG_FILE;
open (LOG_FILE, “>>$LOGFILE”) or do
{
print “Cannot open $LOGFILE\n”;
exit(1);
};
# Uncomment these lines if only interested in specific events
if ($ARGV[0] ne “INSTANCE”) { exit(0); };
#if ($ARGV[0] ne “SERVICEMEMBER”) { exit(0); };
#if ($ARGV[0] ne “SERVICE”) { exit(0); };
#if ($ARGV[0] ne “NODE”) { exit(0); };
for (my $i=0; $i <= $#ARGV; $i++)
{
print LOG_FILE “For Loop $i $ARGV[$i]\n”;
if ($ARGV[$i] =~ m#=#)
{
($key,$value) = (split /=/, $ARGV[$i]);
#print “Key = $key Value = $value\n”;
if ($key eq “service”)
{
$service = $value;
} elsif ($key eq “instance”)
{
$instance = $value;
$ENV{ORACLE_SID} = $value;
} elsif ($key eq “database”)
{
$database = $value;
} elsif ($key eq “host”)
{
$host = $value;
} elsif ($key eq “card”)
{
$card = $value;
} elsif ($key eq “status”)
{
$status = $value;
} elsif ($key eq “reason”)
{
$reason = $value;
}
}
}
print LOG_FILE “Arg=$ARGV[0]\n”;
print LOG_FILE “DB=$database\n”;
print LOG_FILE “Host = $host DB = $database Inst = $instance Service = $service Status = $status Reason = $reason\n”;
# Call relocate service after instance up event is trapped.
#
if ($status eq “up” && $ARGV[0] eq “INSTANCE”)
{
print LOG_FILE “Instance up found. Calling relocate services for $database\n”;
# Call Service relocate
sleep(30);
system(“/usr/local/bin/perl /oracle/scripts/relocate_service.pl $database”);
print LOG_FILE “Success!!!\n”;
}
else
{
print LOG_FILE “Failed: Instance up check failed\n”;
}
Save the below sample script as /oracle/scripts/relocate_service.pl
#!/usr/local/bin/perl
# relocate_service.pl
# This script does a comparison between srvctl config and srvctl status and accordingly relocates the service back based on the config.
my $ret;
my $host1;
# Get dbname passed as an argument from command line
my $dbname;
$argc=scalar @ARGV;
$dbname = $ARGV[0];
chop($host1= `/bin/hostname`);
$ret = system(“srvctl config service -d $dbname|sed ‘s/PREF: //’|sed ‘s/AVAIL:.*\$//’ >/tmp/config_service.log”);
$ret = system(“srvctl status service -d $dbname | sed ‘s/\,\ /\,/g’|cut -f2,7 -d ‘ ‘ | tr -s ‘,’ ‘ ‘>/tmp/status_service.log”);
open(READ_CONFIG, “/tmp/config_service.log”);
open(READ_STATUS, “/tmp/status_service.log”);
my ($rl_from, $rl_to, $no_relocs,$conf_inst_i,$status_inst_i);
# Due to HTML issues, please replace “<" by "” by “>”
while ($config_line = )
{
chomp($config_line);
chop($config_line);
# Due to HTML issues, please replace “<" by "” by “>”
$status_line = ;
chomp($status_line);
if ($config_line eq $status_line)
{
print “$config_line OK\n”;
}
else
{
# Array to store relocate to/from instance names for each service
my @relocate_from;
my @relocate_to;
($conf_serv,@conf_inst) = split / /,$config_line;
($status_serv,@status_inst) = split / /,$status_line;
# Relocate to which node
my $i=0;
my $found;
for (@conf_inst)
{
$conf_inst_i = $_;
$found=0;
for (@status_inst)
{
$status_inst_j = $_;
if ($conf_inst_i eq $status_inst_j) { $found=1; last;}
}
$relocate_to[$i++] = $conf_inst_i if (! $found);
}
# Relocate from which node
my $j=0;
for (@status_inst)
{
$status_inst_i = $_;
$found=0;
for (@conf_inst)
{
$conf_inst_j = $_;
if ($status_inst_i eq $conf_inst_j) { $found=1; last;}
}
$relocate_from[$j++] = $status_inst_i if (! $found);
}
$rl_from= scalar @relocate_from;
$rl_to=scalar @relocate_to;
# How many relocations need to be done
$no_relocs=$rl_from;
if ($rl_from > $rl_to)
{
$no_relocs=$rl_to;
}
# Relocate for all possible instances
if ($no_relocs > 0)
{
for ($i=0; $i<$no_relocs; $i++)
{
$relocate_cmd = “srvctl relocate service -d $dbname -s \”$conf_serv\” -i $relocate_from[$i] -t $relocate_to[$i]”;
$ret = `$relocate_cmd 2>&1`;
print “RELOCATED: $relocate_cmd\n”;
}
}
# Start services on the remaining preferred instanecs
if ($rl_to > $rl_from)
{
for ($i=$no_relocs; $i<$rl_to; $i++)
{
$start_serv_cmd=”srvctl start service -d $dbname -s \”$conf_serv\” -i $relocate_to[$i]”;
$ret = system(“$start_serv_cmd”);
print “STARTED: $start_serv_cmd\n”;
}
}
}
}#End of while
if ($no_relocs > 0)
{
$ret = system(“srvctl status service -d $dbname | sed ‘s/\,\ /\,/g’|cut -f2,7 -d ‘ ‘ | tr -s ‘,’ ‘ ‘>/tmp/status_service_new.log”);
}
Here are the test results. We can see that after an instance is brought back up, the service srv_inst1 is relocated back to it’s preferred instance by the callout script without any manual intervention.
% srvctl config service -d testdb
srv_inst1 PREF: testdb1 AVAIL: testdb2
srv_inst2 PREF: testdb2 AVAIL: testdb1% srvctl status service -d testdb
Service srv_inst1 is running on instance(s) testdb1Service srv_inst2 is running on instance(s) testdb2% srvctl stop instance -d testdb -i testdb1
% srvctl status service -d testdb
Service srv_inst1 is running on instance(s) testdb2Service srv_inst2 is running on instance(s) testdb2% srvctl start instance -d testdb -i testdb1
% srvctl status service -d testdb
Service srv_inst1 is running on instance(s) testdb1Service srv_inst2 is running on instance(s) testdb2
I would thank ritzy .
Thank you
Osama mustafa
# crsctl add css votedisk
# crsctl delete css votedisk
# crsctl add css votedisk -force
# crsctl delete css votedisk -force
$ crsctl query css votedisk
$ dd if= of= bs=4k
OCR BACKUP :
Oracle RAC Taking Backup For OCR By Default , You will Find It On :
cd $ORACLE_BASE/Crs/cdata/jfv_clus
Change the default automatic backup location:
# ocrconfig –backuploc /shared/bak
Back Up OCR Manually :
# ocrconfig –export file name
$ ocrconfig –showbackup
# ocrdump –backupfile file_name
# crsctl stop crs
# ocrconfig –restore <CRS HOME>/cdata/jfv_clus/day.ocr
# crsctl start crs
$ cluvfy comp ocr -n all
Thank You
Osama Mustafa
•Use parallel instance recovery by setting RECOVERY_PARALLISM.•Increase PARALLEL_EXECUTION_MESSAGE_SIZE from its default of 2,148 bytes to 4 KB or 8 KB. This should provide better recovery slave performance.•Set PARALLEL_MIN_SERVERS to CPU_COUNT-1. This will prespawnrecovery slaves at startup time.•Using asynchronous I/O is one of the most crucial factors in recovery time. The first-pass log read uses asynchronous I/O.•Instance recovery uses 50 percent of the default buffer cache for recovery buffers. If this is not enough, some of the steps of instance recovery will be done in several passes. You should be able to identify such situations by looking at your alert.logfile. In that case, you should increase the size of your default buffer cache.