ADDING NEW DISK TO THE EXISTING DISK GROUP/REBALANCING EXISTING DISK GROUP

Things to consider while adding new disk/ Rebalancing existing Disk Group

1)         Make sure the disk are visible on all the cluster nodes; Disk readiness purely depend on the Os & version;

For ex:

Linux 5 – we will be using oracleasm as utility

Linux 6 – we will be using udev utility

2)         When you are planning to add disk to the disk group on the primary database, try to do it when database is having less load.

3)         Increase the ASM power limit to 9 or 10, depending on your environment. Once the rebalancing is completed, bring it to the original value.

Parameter è asm_power_limit

To increase or decrease the asm_power_limit

SQL> alter system set asm_power_limit=9; -- you need to issue it in the ASM instance.

4)         Add disks to one disk group at a time because rebalancing might cause the slowness to the database.

5)         Have a eye on the alert log of the ASM instance. If you are using cluster databases then you need to have a eye on all the instances in the cluster.

6)         You can monitor the rebalancing on the ASM instance using the below query

select group_number, operation, state, power, actual, sofar, est_work, est_rate, est_minutes from gv$asm_operation;

7)         you can check the size of the diskgroup before & after adding the Diskgroup in the ASM.

$ asmcmd –p

ASMCMD> lsdg BHU_B_SYSTEM

8)         Messages in the alert, when the rebalancing starts & when it completes

Disk added on the Node#1 and final status updates on the Node#2

-On Node One

Wed Aug 06 17:09:43 2014

SQL> ALTER DISKGROUP BHU_B_SYSTEM ADD  DISK '/dev/mapper/BHU_B_SYSTEM_05' SIZE 51200M /* ASMCA */

NOTE: GroupBlock outside rolling migration privileged region

NOTE: Assigning number (21,4) to disk (/dev/mapper/BHU_B_SYSTEM_05)

NOTE: requesting all-instance membership refresh for group=21

NOTE: initializing header on grp 21 disk BHU_B_SYSTEM_0004

NOTE: requesting all-instance disk validation for group=21

Wed Aug 06 17:09:45 2014

NOTE: skipping rediscovery for group 21/0xdb38ead3 (BHU_B_SYSTEM) on local instance.

NOTE: requesting all-instance disk validation for group=21

NOTE: skipping rediscovery for group 21/0xdb38ead3 (BHU_B_SYSTEM) on local instance.

NOTE: initiating PST update: grp = 21

Wed Aug 06 17:09:51 2014

GMON updating group 21 at 9546 for pid 73, osid 16865

NOTE: PST update grp = 21 completed successfully

NOTE: membership refresh pending for group 21/0xdb38ead3 (BHU_B_SYSTEM)

GMON querying group 21 at 9547 for pid 18, osid 101970

NOTE: cache opening disk 4 of grp 21: BHU_B_SYSTEM_0004 path:/dev/mapper/BHU_B_SYSTEM_05

Wed Aug 06 17:09:56 2014

NOTE: Attempting voting file refresh on diskgroup BHU_B_SYSTEM

GMON querying group 21 at 9548 for pid 18, osid 101970

SUCCESS: refreshed membership for 21/0xdb38ead3 (BHU_B_SYSTEM)

Wed Aug 06 17:09:56 2014

SUCCESS: ALTER DISKGROUP BHU_B_SYSTEM ADD  DISK '/dev/mapper/BHU_B_SYSTEM_05' SIZE 51200M /* ASMCA */

NOTE: starting rebalance of group 21/0xdb38ead3 (BHU_B_SYSTEM) at power 9

Starting background process ARB0

Wed Aug 06 17:09:56 2014

ARB0 started with pid=74, OS id=91751

NOTE: assigning ARB0 to group 21/0xdb38ead3 (BHU_B_SYSTEM) with 9 parallel I/Os

NOTE: Attempting voting file refresh on diskgroup BHU_B_SYSTEM

Wed Aug 06 17:11:01 2014

-On Second One

NOTE: disk validation pending for group 21/0xdb38cf46 (BHU_B_SYSTEM)

SUCCESS: validated disks for 21/0xdb38cf46 (BHU_B_SYSTEM)

NOTE: disk validation pending for group 21/0xdb38cf46 (BHU_B_SYSTEM)

NOTE: Assigning number (21,4) to disk (/dev/mapper/BHU_B_SYSTEM_05)

SUCCESS: validated disks for 21/0xdb38cf46 (BHU_B_SYSTEM)

NOTE: membership refresh pending for group 21/0xdb38cf46 (BHU_B_SYSTEM)

Wed Aug 06 17:09:54 2014

GMON querying group 21 at 9721 for pid 18, osid 84704

NOTE: cache opening disk 4 of grp 21: BHU_B_SYSTEM_0004 path:/dev/mapper/BHU_B_SYSTEM_05

GMON querying group 21 at 9722 for pid 18, osid 84704

SUCCESS: refreshed membership for 21/0xdb38cf46 (BHU_B_SYSTEM)

Wed Aug 06 17:09:58 2014

NOTE: Attempting voting file refresh on diskgroup BHU_B_SYSTEM

Adding disk

1)         I am using asmca to add the disk to the existing Diskgroup.

2)         You need to plan what kind of redundancy you are opting for, if you are going for a new diskgroup.

3)   As soon as you added the disk in the asmca, you will be shown an output as “Disks added successfully to disk group DISK_GROUP_NAME”. You need to see it in the sql prompt(on ASM instance) to know about the rebalancing status.

Happy Learning :)

OGG-01296 Error mapping from SAP.BHU1000 to SAP_SCHEMA.BHU1000

  OGG-01296  Error mapping from SAP.BHU1000 to SAP_SCHEMA.BHU1000

Situation

We have added new tables to the existing Extract & Replicate. after adding the tables in the replicate, Replicate got aborted with the below error message.

2014-04-17 14:23:19  WARNING OGG-00869  No unique key is defined for table 'BHU1000'. All viable columns will be used to represent the key, but may not guarantee uniqueness.  KEYCOLS may be used to define the key.

2014-04-17 14:23:19  WARNING OGG-01503  Aborting BATCHSQL transaction. Mapping error.

2014-04-17 14:23:19  WARNING OGG-01137  BATCHSQL suspended, continuing in normal mode.

2014-04-17 14:23:19  WARNING OGG-01003  Repositioning to rba 95972561 in seqno 309.

2014-04-17 14:23:19  WARNING OGG-01431  Aborted grouped transaction on 'SAP_SCHEMA.BHU1000', Mapping error.

2014-04-17 14:23:19  WARNING OGG-01003  Repositioning to rba 95972561 in seqno 309.

2014-04-17 14:23:19  WARNING OGG-01151  Error mapping from SAP.BHU1000 to SAP_SCHEMA.BHU1000.

2014-04-17 14:23:19  WARNING OGG-01003  Repositioning to rba 95972561 in seqno 309.

Source Context :

  SourceModule            : [er.errors]

  SourceID                : [/scratch/aime1/adestore/views/aime1_adc4150256/oggcore/OpenSys/src/app/er/errors.cpp]

  SourceFunction          : [take_rep_err_action]

  SourceLine              : [632]

  ThreadBacktrace         : [8] elements

                          : [/oracle/product/ggs_112105/CRIC1/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x2b83a93900de]]

                     : [/oracle/product/ggs_112105/CRIC1/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x2cc) [0x2b83a9388dfc]]

                          : [/oracle/product/ggs_112105/CRIC1/libgglog.so(_MSG_ERR_MAP_TO_TANDEM_FAILED(CSourceContext*, ggs::gglib::ggapp::CQualDBObjName<(DBObjType)1

> const&, ggs::gglib::ggapp::CQualDBObjName<(DBObjType)1> const&, CMessageFactory::MessageDisposition)+0x53) [0x2b83a9384cc7]]

                          : [/oracle/product/ggs_112105/CRIC1/replicat(take_rep_err_action(short, int, char const*, extr_ptr_def*, __std_rec_hdr*, char*, file_def*, bo

ol)+0xaa5) [0x520cc1]]

                          : [/oracle/product/ggs_112105/CRIC1/replicat(process_extract_loop()+0x2e42) [0x53ad82]]

                          : [/oracle/product/ggs_112105/CRIC1/replicat(main+0x732) [0x54bf02]]

                          : [/lib64/libc.so.6(__libc_start_main+0xf4) [0x3cd661d9c4]]

                          : [/oracle/product/ggs_112105/CRIC1/replicat(__gxx_personality_v0+0x31a) [0x4c035a]]

2014-04-17 14:23:19  ERROR   OGG-01296  Error mapping from SAP.BHU1000 to SAP_SCHEMA.BHU1000.

FINDING & SOLUTION

1)      We found that table doesn’t have any primary key columns

2)      We found that table has 4 indexes in the source side (extract) but no indexes created on the replicate side while adding tables in the replicate.

3)      We found multiple check constraints are created but no primary key constraints.

We have created all the indexes on the target side (Replicat database) for the table(similar one like source) and try to start the replicate.

We found apply process goes fine and that solves the problem.

Happy reading JJJ

Error: ORA-16651: requirements not met for enabling fast-start failover

When I try to enable the fast start failover in the data guard below I got the below error message

Error: ORA-16651: requirements not met for enabling fast-start failover

DGMGRL> show configuration;

Configuration - DG_BHU

  Protection Mode: MaxAvailability

  Databases:

    BHU_A - Primary database

    BHU_B - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS

DGMGRL> show fast_start failover;

Fast-Start Failover: DISABLED

  Threshold:        90 seconds

  Target:           (none)

  Observer:         Bhuvan_observer

  Lag Limit:        30 seconds

  Shutdown Primary: TRUE

  Auto-reinstate:   TRUE

Configurable Failover Conditions

  Health Conditions:

    Corrupted Controlfile          YES

    Corrupted Dictionary           YES

    Inaccessible Logfile            NO

    Stuck Archiver                  NO

    Datafile Offline               YES

  Oracle Error Conditions:

    (none)

DGMGRL> enable fast_start failover;

Error: ORA-16651: requirements not met for enabling fast-start failover

Failed.

SOLUTION

1)  Check whether the flashback database is enabled on primary & standby database

Select flashback_on from gv$database;

2) Check whether you have set the FastStartFailoverTarget parameter.

Ex:

 edit database 'BHU_A' set property FastStartFailoverTarget='BHU_B';

edit database 'BHU_B' set property FastStartFailoverTarget='BHU_A';

RMAN-05541: no archived logs found in target database

This is very interesting error message while performing RMAN Duplication of full database

I have taken a full database backup in a local disk including archive log; when i try to restore the production database in the test environment it has thrown below error message.

ERROR MESSAGE DURING RMAN DUPLICATION

RMAN-01005: Errors in memory script
RMAN-01005: RMAN-03015: error occurred in stored script Memory Script
RMAN-06136: ORACLE error from auxiliary database: ORA-01507: database not mounted
ORA-06512: at "SYS.X$DBMS_RCVMAN", line 13466
ORA-06512: at line 1
RMAN-05541: no archived logs found in target database
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================RMAN-03002: failure of Duplicate Db command at 08/06/2013 13:56:37
RMAN-05501: aborting duplication of target database

BACKUP START AT: 13:00.00 and completed at 15:30. RMAN looking for the archive log backup files from 12:50:00 to 13.10.00(I have supplied 10 min before & after the backup)

SUGGESTION#1
CHECK THE BACKUP FILES PERMISSION AS A FIRST THINGS

SUGGESTION#2
1) Perform switch log file before start the backup

2) After a long struggle; we found that RMAN is looking for some old archive logs during the start of the hot backup; once we place the old archive log backup in the backup area then RMAN duplication script started working fine. i doesn’t know why RMAN expecting the old archive log files before the backup start; that how it is working in oracle 11gR2(11.2.0.3.0)

ORA-16795: the standby database needs to be re-created

DB CONFIGURATION

2 node primary database

2 node standby database

Fast_Start FailOver(observer)- Enabled

BHU_A è Primary database

BHU_B è standby database

n  Due to the network issues, FSFO failover the standby database (BHU_B) as a new primary and old primary database (BHU_A) is in the mount stage.

oracle BHU bhuora001> dgmgrl

DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.

DGMGRL> connect sys /xxxxxxxxxxxxxx

Connected.

DGMGRL> show configuration

Configuration - DG_BHU

  Protection Mode: MaxAvailability

  Databases:

    BHU_B - Primary database

      Warning: ORA-16817: unsynchronized fast-start failover configuration

    BHU_A - (*) Physical standby database (disabled)

      ORA-16795: the standby database needs to be re-created

Fast-Start Failover: ENABLED

Configuration Status:

WARNING

n  We check the flashback enabled option on both old & new primary. We are showing the output of new primary database

SQL> select flashback_on from v$database;

FLASHBACK_ON

------------------

YES

n  On the NEW PRIMARY SITE; check when the conversion happened(standby to primary) and you can see the time by looking to scn_to_timestamp conversion.

SQL> SELECT TO_CHAR(STANDBY_BECAME_PRIMARY_SCN) FROM V$DATABASE;

TO_CHAR(STANDBY_BECAME_PRIMARY_SCN)

----------------------------------------

12788498729592

SQL> SELECT SCN_TO_TIMESTAMP(12788498729592) from dual;

SCN_TO_TIMESTAMP(12788498729592)

---------------------------------------------------------------------------

08-AUG-13 22.04.00.000000000

On the OLD PRIMARY DATABASE(BHU_A). Check the status of database

SQL> select name,open_mode,database_role from gv$database;

NAME      OPEN_MODE            DATABASE_ROLE

--------- -------------------- ----------------

BHU    MOUNTED              PRIMARY

SQL> select flashback_on from v$database;

FLASHBACK_ON

------------------

YES

n When we try to check  CURRENT_SCN number on the old primary and it has displayed as “0”

SQL> select current_scn from v$database;

CURRENT_SCN

-----------

          0

Now, We are converting the old primary database as a NEW STANDBY DATABASE

n  We are mounting the database first

SQL> shutdown immediate;

ORA-01109: database not open

Database dismounted.

ORACLE instance shut down.

SQL> startup mount;

ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance

ORACLE instance started.

Total System Global Area 2271580160 bytes

Fixed Size                  2230352 bytes

Variable Size            1191184304 bytes

Database Buffers         1073741824 bytes

Redo Buffers                4423680 bytes

Database mounted.

n  We have identified the failover time from the new database. so we will use that time to flashback the old primary database; we have add 20 further to actual time.

SQL> FLASHBACK DATABASE TO TIMESTAMP to_timestamp('2013-08-08 21:50:00', 'YYYY-MM-DD HH24:MI:SS');

Flashback complete.

n  We are checking the current role before converting the old primary to new standby database.

SQL> select name,open_mode,database_role from v$database;

NAME      OPEN_MODE            DATABASE_ROLE

--------- -------------------- ----------------

BHU    MOUNTED              PRIMARY

n  Converting database to standby database.

SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY;

Database altered.

n  We are checking the current role after converting the old primary to new standby database.

SQL> select name,open_mode,database_role from v$database;

select name,open_mode,database_role from v$database

                                         *

ERROR at line 1:

ORA-01507: database not mounted

n  After converting the database; database will go to nomount stage; so we have to stop & start the database.

SQL> shutdown immediate;

ORA-01507: database not mounted

ORACLE instance shut down.

SQL> startup mount;

ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance

ORACLE instance started.

Total System Global Area 2271580160 bytes

Fixed Size                  2230352 bytes

Variable Size            1191184304 bytes

Database Buffers         1073741824 bytes

Redo Buffers                4423680 bytes

Database mounted.

SQL> select name,open_mode,database_role from v$database;

NAME      OPEN_MODE            DATABASE_ROLE

--------- -------------------- ----------------

BHU    MOUNTED              PHYSICAL STANDBY

n  We could see the current status of the old primary as new standby database.

n  Now we are login to the dg broker to enable the new standby database

DGMGRL> enable database 'BHU_A';

Enabled.

n  Once enabled, oracle will take some time to apply the logs till the current time

DGMGRL> show configuration;

Configuration - DG_BHU

  Protection Mode: MaxAvailability

  Databases:

    BHU_B - Primary database

      Warning: ORA-16817: unsynchronized fast-start failover configuration

    BHU_A - (*) Physical standby database

      Warning: ORA-16817: unsynchronized fast-start failover configuration

Fast-Start Failover: ENABLED

Configuration Status:

WARNING

n  Now primary database & standby database are in sync.

DGMGRL> show configuration;

Configuration - DG_BHU

  Protection Mode: MaxAvailability

  Databases:

    BHU_B - Primary database

    BHU_A - (*) Physical standby database

Fast-Start Failover: ENABLED

Configuration Status:

SUCCESS