Tuesday, December 20, 2011

state:blocked in the listener service view

PROBLEM
When I try to log in to the listener service for a particular listener, I found the state has been specified as BLOCKED.


oracle> lsnrctl service LISTENER_SCAN1

LSNRCTL for Linux: Version 11.2.0.2.0 - Production on 20-DEC-2011 13:13:13

Copyright (c) 1991, 2010, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
Services Summary...
Service "BHU_A" has 1 instance(s).
  Instance "BHU_2", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:blocked
         REMOTE SERVER
         (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=bhuora02-vip)(PORT=1532)))
Service "BHU_B" has 2 instance(s).
  Instance "BHU_1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=bhuora01-vip)(PORT=1532)))
  Instance "BHU_2", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:1 refused:0 state:ready
         REMOTE SERVER
         (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=bhuora02-vip)(PORT=1532)))

POSSIBLE CAUSE OF THIS ISSUE

1) When the listener believes the current number of connections has reached maximum load, it may set the state of the service handler for an instance to "blocked" and begin refusing incoming client connections with either of the following errors:
 
Below error will be appeared on the listener log
TNS-12516 , TNS-12519 , TNS-12520
Sol: check the select * from gv$resource_limit;
Check for the sessions & processes, whether it has reached the MAX_UTILIZATION & CURRENT_UTILIZATION. It is nearing to the allocate value then try to increase the sessions & processes values for the database.
 
2) In RAC scenarios, if any one modifies(put a incorrect)  either primary instance or standby instance listener parameters like local_listener,remote_listener & listener_networks. Then primary database listener might registries in the standby database – vice versa then it will place the state as BLOCKED.
Sol: verify all the listener parameters & ports are property specified for each instance in the cluster database
 
 

Monday, December 19, 2011

ORA-15027: active use of diskgroup "TEST_RENAME" precludes its dismount


When I try to umount the diskgroup from the ASM, it was showing the below error


I have shutdown the database using srvctl, but the database is not shutdown properly(I come to know after analyzing it) . I could see that cluster resource says that the database is down, so I didn’t  check database further. Then I found the list of active file on the DISKGROUP using “lsof” option.

Trying to umount the diskgroup

oracle +ASM1> asmcmd umount TEST_RENAME
ORA-15032: not all alterations performed
ORA-15027: active use of diskgroup "TEST_RENAME" precludes its dismount (DBD ERROR: OCIStmtExecute)

NOTE# Check the active files in ASM using the lsof. When your DISKGROUP is in active stage, we cant umount or remove the DISKGROUP. You can check by v$asm_client option but the below option is very simple.

ASMCMD [+] > lsof
DB_Name  Instance_Name  Path
….(REMOVED OTHER FILES)
BHU_B    ZS2_1          +test_rename/BHU_b/datafile/test.256.769969469

Thursday, December 15, 2011

Changing apply instance using dgmgrl


I want to move my apply instance from BHU_1 to BHU_2 on BHU_A database. I have been using a different approach and you need to connect to multiple sessions/servers to perform this task. I found an easiest way to change it.

Please find the current status of the apply instance.

DGMGRL> show database 'BHU_A';

Database - BHU_A

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: OFF
  Instance(s):
    BHU_1 (apply instance)
    BHU_2

Database Status:
SUCCESS

If we want move the apply instance, we need to stop the recover process. So I am stopping the recover process on the node#1(BHU_1)

DGMGRL> edit database 'BHU_A' set state='APPLY-OFF';
Succeeded.

Enabling recover process and asking the system to start the apply process on the node#2

DGMGRL> edit database 'BHU_A' set state='APPLY-ON' with apply instance ='BHU_2';
Succeeded.

Now let us look the status of apply instance

DGMGRL> show database 'BHU_A';

Database - BHU_A

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: OFF
  Instance(s):
    BHU_1
    BHU_2 (apply instance)

Database Status:
SUCCESS

Checking the status of the data guard.

DGMGRL> show configuration;

Configuration - DG_BHU

  Protection Mode: MaxAvailability
  Databases:
    BHU_B - Primary database
    BHU_A - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

Happy learning!!!!!!!!

Reset/Unset Oracle parameters


A parameter value can be reset to the default using the following command

Note: Do not try to change the value of a parameter with” (or) (‘) or (null) to reset it to default
Below example shows, how to reset a oracle parameter. I am running this test on oracle 11gR2(11.2.0.2.0)


SQL> show parameter convert

NAME                                 TYPE        VALUE
------------------------------------ ----------- --------------------
db_file_name_convert                 string   +DATA1,+DG_DATA1
log_file_name_convert                string   +LOG,+DG_LOG


SQL> alter system reset db_file_name_convert scope=spfile sid='*';

System altered.

SQL> alter system reset log_file_name_convert scope=spfile sid='*';

System altered.

Resetting with SCOPE=MEMORY or SCOPE=BOTH is currently not supported. If you proceeding with the parameter change, you will get an error message as below


SQL> alter system reset db_file_name_convert scope=both sid='*';
alter system reset db_file_name_convert scope=both sid='*'
*
ERROR at line 1:
ORA-32029: resetting with SCOPE=MEMORY or SCOPE=BOTH is currently not supported


I tried to change a dynamic parameter which can changed during the run time, even those parameter are not supported in “RESET” with memory or both option

SQL> alter system set log_archive_dest_state_3=DEFER scope=both sid='*';

System altered.

SQL> show parameter log_archive_dest_state_3

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_state_3             string      DEFER

SQL> alter system reset log_archive_dest_state_3 scope=both sid='*';
alter system reset log_archive_dest_state_3 scope=both sid='*'
*
ERROR at line 1:
ORA-32029: resetting with SCOPE=MEMORY or SCOPE=BOTH is currently not supported


SQL> alter system set log_archive_dest_state_3=ENABLE scope=both sid='*';

System altered.

I am able to change the parameter with out bouncing the database.

SQL> show parameter log_archive_dest_state_3

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_state_3             string      ENABLE

Wednesday, December 14, 2011

CREATE DISKGROUP MANUAL in 11gR2


In the document I am creating DISK and creating the DISKGROUP in the cluster environment using manual method.  I have a two node primary database & two node standby database.

We need to identify the raw disks which are candidates for the ASM storage. I have check in the primary & standby site

Below disk are free on both side (primary RAC DB & Standby RAC DB)
raw_vote_03    
raw_vote_04    
raw_vote_05    


CREATING ORACLEASM DISKS using ‘oracleasm createdisk’ command as root. This command will make the disks available to ASMLib. List the existing disks using the listdisks option to verify that ASMLib is aware of the marked new disks.


[root@bhuora01~]# oracleasm createdisk DG_TEST /dev/mapper/raw_vote_03
Writing disk header: done
Instantiating disk: done
[root@bhuora01]# oracleasm createdisk DG_TEST1 /dev/mapper/raw_vote_04
Writing disk header: done
Instantiating disk: done
[root@bhuora01]# oracleasm createdisk DG_TEST2 /dev/mapper/raw_vote_05
Writing disk header: done
Instantiating disk: done

VERIFYING ASK DISK LABEL BY PROVIDING DISK

[root@bhuora01 ~]# oracleasm querydisk /dev/mapper/raw_vote_03
Device "/dev/mapper/raw_vote_03" is marked an ASM disk with the label "DG_TEST"
[root@bhuora01 ~]# oracleasm querydisk /dev/mapper/raw_vote_04
Device "/dev/mapper/raw_vote_04" is marked an ASM disk with the label "DG_TEST1"
[root@bhuora01 ~]# oracleasm querydisk /dev/mapper/raw_vote_05
Device "/dev/mapper/raw_vote_05" is marked an ASM disk with the label "DG_TEST2"

VERIFYING DISK USING listdiks

[root@bhuora01 ~]# oracleasm listdisks
DG_TEST
DG_TEST1
DG_TEST2


[root@bhuora01 ~]# ls -alrt /dev/oracleasm/disks/DG*
total 0
drwxr-xr-x 4 root   root         0 Nov 22 13:14 ..
drwxr-xr-x 1 root   root         0 Nov 22 13:14 .
brw-rw---- 1 oracle asmdba 253, 26 Dec 13 15:36 DG_TEST
brw-rw---- 1 oracle asmdba 253, 27 Dec 13 15:36 DG_TEST1
brw-rw---- 1 oracle asmdba 253, 28 Dec 13 15:36 DG_TEST2


Make ASM disks available to other nodes in the cluster by running the scandisk command. This needs to be run on all other nodes.

[root@bhuora02 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "DG_TEST"
Instantiating disk "DG_TEST1"
Instantiating disk "DG_TEST2"
[root@bhuora02 ~]#

You can see from the scandisks commands, it display above 3 disks are creating newly on the first node

[root@bhuora02 ~]# ls -alrt /dev/oracleasm/disks/DG*
brw-rw---- 1 oracle asmdba 253, 36 Dec 13 15:40 /dev/oracleasm/disks/DG_TEST
brw-rw---- 1 oracle asmdba 253, 38 Dec 13 15:40 /dev/oracleasm/disks/DG_TEST1
brw-rw---- 1 oracle asmdba 253, 40 Dec 13 15:40 /dev/oracleasm/disks/DG_TEST2


CREATING THE DISK GROUP IN COMMAND PROMPT

oracle +ASM1 bhuora02> sqlplus / as sysasm

SQL*Plus: Release 11.2.0.2.0 Production on Tue Dec 13 16:02:22 2011

Copyright (c) 1982, 2010, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> CREATE DISKGROUP TEST EXTERNAL REDUNDANCY
DISK '/dev/oracleasm/disks/DG_TEST',
'/dev/oracleasm/disks/DG_TEST1',
'/dev/oracleasm/disks/DG_TEST2' ATTRIBUTE 'au_size'='4M',
'compatible.asm' = '11.2.0.2.0',
'compatible.rdbms' = '11.2.0.2.0',
'compatible.advm' = '11.2.0.2.0';

Diskgroup created.


SQL> select * from v$asm_diskgroup where name='TEST';

GROUP_NUMBER NAME                           SECTOR_SIZE BLOCK_SIZE
------------ ------------------------------ ----------- ----------
ALLOCATION_UNIT_SIZE STATE       TYPE     TOTAL_MB    FREE_MB HOT_USED_MB
-------------------- ----------- ------ ---------- ---------- -----------
COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
------------ ----------------------- -------------- -------------
COMPATIBILITY
------------------------------------------------------------
DATABASE_COMPATIBILITY                                       V
------------------------------------------------------------ -
          32 TEST                                   512       4096
             4194304 MOUNTED     EXTERN       6144       6004           0
         140                       0           6004             0

GROUP_NUMBER NAME                           SECTOR_SIZE BLOCK_SIZE
------------ ------------------------------ ----------- ----------
ALLOCATION_UNIT_SIZE STATE       TYPE     TOTAL_MB    FREE_MB HOT_USED_MB
-------------------- ----------- ------ ---------- ---------- -----------
COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
------------ ----------------------- -------------- -------------
COMPATIBILITY
------------------------------------------------------------
DATABASE_COMPATIBILITY                                       V
------------------------------------------------------------ -
11.2.0.2.0
11.2.0.2.0                                                   N


ON NODE#2 if you run the same queries on the second node, it shows there is no disk group with the name. When I check further on the issue, I found that we need to manual mount in all the cluster nodes.

SQL> select * from v$asm_diskgroup where name='TEST';

GROUP_NUMBER NAME                           SECTOR_SIZE BLOCK_SIZE
------------ ------------------------------ ----------- ----------
ALLOCATION_UNIT_SIZE STATE       TYPE     TOTAL_MB    FREE_MB HOT_USED_MB
-------------------- ----------- ------ ---------- ---------- -----------
COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
------------ ----------------------- -------------- -------------
COMPATIBILITY
------------------------------------------------------------
DATABASE_COMPATIBILITY                                       V
------------------------------------------------------------ -
           0 TEST                                     0       4096
                   0 DISMOUNTED                  0          0           0
           0                       0              0             0

GROUP_NUMBER NAME                           SECTOR_SIZE BLOCK_SIZE
------------ ------------------------------ ----------- ----------
ALLOCATION_UNIT_SIZE STATE       TYPE     TOTAL_MB    FREE_MB HOT_USED_MB
-------------------- ----------- ------ ---------- ---------- -----------
COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
------------ ----------------------- -------------- -------------
COMPATIBILITY
------------------------------------------------------------
DATABASE_COMPATIBILITY                                       V
------------------------------------------------------------ -
0.0.0.0.0
0.0.0.0.0                                                    N

WHEN I TRY TO DISMOUNT DISK GROUP, I GOT AN ERROR SAYING THAT THERE IS NO DISK AVAILABLE ON THE NODE

SQL> alter diskgroup test dismount;
alter diskgroup test dismount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15001: diskgroup "TEST" does not exist or is not mounted

THEN MOUNT THE DISK GROUP

SQL> alter diskgroup test mount;

Diskgroup altered.

SQL> select * from v$asm_diskgroup where name='TEST';

GROUP_NUMBER NAME                           SECTOR_SIZE BLOCK_SIZE
------------ ------------------------------ ----------- ----------
ALLOCATION_UNIT_SIZE STATE       TYPE     TOTAL_MB    FREE_MB HOT_USED_MB
-------------------- ----------- ------ ---------- ---------- -----------
COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
------------ ----------------------- -------------- -------------
COMPATIBILITY
------------------------------------------------------------
DATABASE_COMPATIBILITY                                       V
------------------------------------------------------------ -
          32 TEST                                   512       4096
             4194304 MOUNTED     EXTERN       6144       6004           0
         140                       0           6004             0

GROUP_NUMBER NAME                           SECTOR_SIZE BLOCK_SIZE
------------ ------------------------------ ----------- ----------
ALLOCATION_UNIT_SIZE STATE       TYPE     TOTAL_MB    FREE_MB HOT_USED_MB
-------------------- ----------- ------ ---------- ---------- -----------
COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
------------ ----------------------- -------------- -------------
COMPATIBILITY
------------------------------------------------------------
DATABASE_COMPATIBILITY                                       V
------------------------------------------------------------ -
11.2.0.2.0
11.2.0.2.0                                                   N


SQL>

NOTE#

I have the DISK GROUP is automatically added in the cluster registry.

oracle +ASM1 bhuora02> crsctl stat res ora.TEST.dg -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS     
----------------------------------------------------------------------------
Local Resources
----------------------------------------------------------------------------
ora.TEST.dg
               ONLINE  ONLINE       bhuora02                                
               ONLINE  ONLINE       bhuora01                                

Monday, December 12, 2011

FAILOVER Database with out DG Broker

           MANUAL FAILOVER ON PHYSICAL STANDBY DATABASE
 
This note describes how to perform manual failover operations in physical standby databases with out DG Broker.
 If you are performing for RAC, then you can perform with the single instance and keep other instance in the 
shutdown mode.
 
After a Failover, the Standby Database becoming a Primary now, cannot switchback to become a 
Standby Database again. So you can perform Failover only once.
 
A Failover can be performed when all or most of the information until the Unavailability of the Primary Database
 was propagated to the Standby. The usage of Standby Redo Logs is a great advantage here. If you have no
 Standby Redo Logs available, you will always encounter some Data Loss (depending on the Changes since
 the latest Log Switch). To perform a Failover just follow these steps:
 
STEP#1 The Primary Database is down for any reason
 
Verify a Standby Redo Log is in use for Primary current Online Redo Log. You then find in the ALERT.LOG
of the Standby something like: RFS:  If this is the case run the following commands:
   
  SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
  
  This cancels the normal managed Recovery. 
 
STEP#2 To get the Standby Redo Log Information is still required. Therefore issue this command:
 
   SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH;
   
 
If you have the standby redo log then go to STEP 3
 
  2 (a) If a Standby Redo Log is not used for any reason and then run this one:
 
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH
 
 SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH SKIP STANDBY LOGFILE;
 
Please keep in mind that this one causes (Minimal) Data Loss as the latest  information from the down Primary
 Database is not available anymore. 
 
Once this is complete (This performs a complete Recovery or incomplete   Recovery until the last SCN
 included in the latest archived Log available at the Standby), you can now make the Standby Database a 
Primary:
  
STEP #3 converting standby role to the primary role
 
 
SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY;

Database altered.
   
WARNING: This will only succeed if the correct RECOVER FINISH-statement was issued before.
 If you forgot the 'SKIP STANDBY LOGFILE' although you have no Standby RedoLogs, the COMMIT
 to Switchover will fail with the error that more Media Recovery is required here.
 
If above statement fail then perform the below step else go for STEP 4
 
STEP (3A)
If the COMMIT TO SWITCHOVER fails for any reason you have to use the ACTIVATE command 
which forces the Failover (and may cause Data Loss !!)
  
   SQL> ALTER DATABASE ACTIVATE STANDBY DATABASE;
 
STEP # 4 Shutdown and restart the Database after this command ended successfully:
 
   SQL> SHUTDOWN IMMEDIATE;
   SQL> STARTUP
   
- Now the Standby is open as a new Primary Database 

Happy learning!!!!

Wednesday, December 7, 2011

RESTORING OCR DISK & VOTING DISK ON ASM DISK in 11gR2


When we have lost or having a corruption issue on the OCR & VOTING Disk, we have follow the below procedure to bring it back.

BASIC CHECKS : To Check whether ocrcheck is corrupted or not

# ocrcheck

 

When using an ASM disk group for CRS there are typically 3 different types of files located in the OCR disk group that potentially need to be restored/recreated for function of the cluster.
  • Oracle Cluster Registry file (OCR)
  • Voting files
  •  Shared SPFILE for the ASM instances
In the below condition, you are not able to start the ASM instance because of the corruption issue

Step #1 Stop cluster on each node (Root user).

# crsctl stop crs -f

Step #2 we are starting the cluster in the excusive mode (Root user)

As root start GI in exclusive mode on one node only:

In 11201 RAC, we have to use below option to start the cluster in the exclusive mode.
# crsctl start crs -excl

In 11202 RAC, we have to use below option to start the cluster in the exclusive mode.
# crsctl start crs -excl -nocrs

Note: A new option '-nocrs' has been introduced with  11.2.0.2, which prevents the start of the ora.crsd resource. It is vital that this option is specified; otherwise the failure to start the ora.crsd resource will tear down ora.cluster_interconnect.haip, which in turn will cause ASM to crash.


If you don’t have the OCR DISK GROUP, then we need to create the disk group else move to restoring OCR DISK


Step #3 RESTORING OCR RESTORE

To Know the OCR Location on the cluster environment
$ cat /etc/oracle/ocr.loc  -- In Linux

To Check whether ocrcheck is corrupted or not

# ocrcheck

Check whether ocrcheck is able to complete it successfully

OCR CHECK Ex
# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       4404
         Available space (kbytes) :     257716
         ID                       : 1306201859
         Device/File Name         :  +OCR_VOTE
                                    Device/File integrity check succeeded
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
         Cluster registry integrity check succeeded

         Logical corruption check succeeded
        

Note: 1) Check whether cluster registry integrity check is successful.
          2) When you run as root user, logical corruption check will be bypassed.
If you run as oracle user, you can see this line end of the “ocrcheck” output.
“Logical corruption check bypassed due to non-privileged user”


To Know the OCR Location on the cluster environment
$ cat /etc/oracle/ocr.loc  -- In Linux
If the OCR DISK corrupted, then perform the below steps

Locate OCR LOG file location
$GRID_HOME /log/<hostname>/client/ocrcheck_<pid>.log
Locate the latest automatic OCR backup
$GRID_HOME\bin\ocrconfig –showbackup

Restore the latest OCR backup(root user)
# ocrconfig -restore $GRID_HOME/cdata/bhurac/backup00.ocr
bhurac è SCAN NAME for the cluster

Step #4 VOTING DISK RECREATE
           
Recreate the Voting file (root user)
The Voting file needs to be initialized in the CRS disk group
# crsctl replace votedisk +OCR_DISK
Note: 1) Above command will Re-create/move your voting disk in the specified ASM Disk Group, if you query the voting disk it will display your voting disk location in the DISK Group which has been specified above.
2)  Voting File is  no longer supported to take a manual backup of it with dd.  Instead, the Voting File gets backed up automatically into the OCR.

Query Voting Disk location

# $GRID_HOME/bin/crsctl query css votedisk

Note: You cannot create more than 1 voting disk in the same or on another/different Disk group disk when using External Redundancy in 11.2. The rules are as follows:
External = 1 voting disk
Normal= 3 voting disk
High= 5 voting disk

NOTE:  IN THIS SCNERIO, ONLY MY OCR & VOTING IS CORRUPED. IN ANOTHER POST, I WILL PROVIDE A STEPS TO RESTORE SPFILE. WHEN THERE IS NO COPY OF IT.

Step #5 Stop & start the cluster

Shutdown CRS è CRS is running in exclusive mode, it needs to be shutdown (Root User).

# crsctl stop crs -f

Start CRS è Start the CRS in one node, if everything is ok then start the CRS in other nodes (root user).

# crsctl start crs

CRS Status è Once started, you can check the status of the CRS(Root / Oracle user)

# crsctl stat res –t –init      à if you are checking for one node
# crsctl check cluster –all  à if you are checking for entire cluster.
# $GRID_HOME/bin/crsctl status resource –t  à Gives details information about each resource.

Important Tips

1) Oracle Clusterware 11g Release 2 backs up the OCR automatically every four hours on a schedule that is dependent on when the node started
  • 4-hour backups (3 max) –backup00.ocr, backup01.ocr, and backup02.ocr.
  • Daily backups (2 max) – day.ocr and day_.ocr
  • Weekly backups (2 max) – week.ocr and week_.ocr
You can use the ocrconfig command to view the current OCR backups as seen in this
Ocrconfig –showbackup auto
 
Note: automatic backups will not occur, when the cluster is down
2) Verifying OCR integrity of entire cluster nodes by running CVU command:
$ cluvfy comp ocr -n all -verbose

3) Oracle Local Repository (OLR), this repository designed to store information and profiles for local resources, resources that dedicated to particular node. It improves the performance of accessing local resources profile information, redundancy and manageability. In Grid Infrastructure RAC configuration there is usual one global shared OCR and OLR’s on each node. In Oracle Restart environment there is only OLR repository. In 11g R2 there is also new feature, Grid Plug and Play (GPNP), as the name implies, it helps to automate and simplify some of the aspects of grid administration. GPNP maintains profile, it is XML file that stores the configuration information of some components maintained by GPNP, for example vips and interconnect information is stored here.

HAPPY LEARNING!!!!!!!!!