Sunday, 13 November 2016

ORA-12012: error on auto execute of job “SYS”.”BSLN_MAINTAIN_STATS_JOB”


Symptoms

ORA-12012: error on auto execute of job “SYS”.”BSLN_MAINTAIN_STATS_JOB”

Errors In Alert Log File


/u01/app/oracle/diag/rdbms/ORADB/ORADB/trace/ORADB_j000_7733378.trc:
ORA-12012: error on auto execute of job “SYS”.”BSLN_MAINTAIN_STATS_JOB”
ORA-06502: PL/SQL: numeric or value error
ORA-06512: at “DBSNMP.BSLN_INTERNAL”, line 2073
ORA-06512: at line 1
Sat Mar 24 23:01:18 2012

Have a look at trace file


*** 2012-03-24 23:00:13.943
*** SESSION ID:(23.6871) 2012-03-24 23:00:13.943
*** CLIENT ID:() 2012-03-24 23:00:13.943
*** SERVICE NAME:(SYS$USERS) 2012-03-24 23:00:13.943
*** MODULE NAME:(DBMS_SCHEDULER) 2012-03-24 23:00:13.943
*** ACTION NAME:(BSLN_MAINTAIN_STATS_JOB) 2012-03-24 23:00:13.943

ORA-12012: error on auto execute of job “SYS”.”BSLN_MAINTAIN_STATS_JOB”
ORA-06502: PL/SQL: numeric or value error
ORA-06512: at “DBSNMP.BSLN_INTERNAL”, line 2073
ORA-06512: at line 1

Cause


Table DBSNMP.BSLN_BASELINES contains inconsistent information that is causing the job to fail.

Check the job status

SQL> select log_date,status from dba_scheduler_job_run_details where job_name=’BSLN_MAINTAIN_STATS_JOB’;

LOG_DATE STATUS
------------------------------------------
10-MAR-12 11.00.13.501250 PM +00:00 FAILED
03-MAR-12 11.00.10.603238 PM +00:00 FAILED
17-MAR-12 11.00.14.793987 PM +00:00 FAILED
24-MAR-12 11.00.13.980545 PM +00:00 FAILED

Role of BSLN_MAINTAIN_STATS_JOB


This job runs the BSLN_MAINTAIN_STATS_PROG program on the BSLN_MAINTAIN_STATS_SCHED schedule.  The program BSLN_MAINTAIN_STATS_PROG will keep the default baseline’s statistics up-to-date

Check the DBSNMP.BSLN_BASELINES table


SQL> select * from DBSNMP.BSLN_BASELINES;

DBID INSTANCE_NAME BASELINE_ID BSLN_GUID TI A STATUS LAST_COMP
-------------------------------------------------------------------------
1166314350 FTEST 0 4AC774574F6C7D60D4ADF390356098C1 NX Y ACTIVE 27-NOV-10
1166314350 TEST 0 CEF9782DC11EE02FCB9E1D821B49FB3F HX Y ACTIVE 27-NOV-10

So on our database we should just have the entry for TEST so we need to delete the entry for FTEST.

Delete the inconsistent entry


SQL> delete from DBSNMP.BSLN_BASELINES where INSTANCE_NAME=’FTEST’;

1 row deleted.

SQL> select * from DBSNMP.BSLN_BASELINES;

DBID INSTANCE_NAME BASELINE_ID BSLN_GUID TI A STATUS LAST_COMP
---------------------------------------------------------------
1166314350 TEST 0 CEF9782DC11EE02FCB9E1D821B49FB3F HX Y ACTIVE 27-NOV-10

Now re-run the job.


SQL> exec dbms_scheduler.run_job(‘BSLN_MAINTAIN_STATS_JOB’,false);

PL/SQL procedure successfully completed.

The job has now successfully run.

SQL> select log_date,status from dba_scheduler_job_run_details where job_name=’BSLN_MAINTAIN_STATS_JOB’;

LOG_DATE STATUS
---------------------------------------------
10-MAR-12 11.00.13.501250 PM +00:00 FAILED
29-MAR-12 01.11.43.054124 PM +01:00 SUCCEEDED
03-MAR-12 11.00.10.603238 PM +00:00 FAILED
17-MAR-12 11.00.14.793987 PM +00:00 FAILED
24-MAR-12 11.00.13.980545 PM +00:00 FAILED

Saturday, 12 November 2016

Snapshot Standby Database-Read Write Mode

The only requirement to have the snapshot standby is that FRA (Flash Recovery Area) must be configured on physical standby database. It is not necessary to have flashback enabled. Below are the steps on how to convert a physical standby database to a snapshot standby database and viceversa


Oracle Database version: 11.2.0.3 Enterprise Edition
Primary database: DCPROC

Details with respect to the primary database:
      
SQL> select status,instance_name,database_role,open_mode from v$database,v$Instance;

STATUS       INSTANCE_NAME    DATABASE_ROLE    OPEN_MODE
------------ ---------------- ---------------- --------------------
OPEN         srprim           PRIMARY          READ WRITE

SQL> select thread#,max(sequence#) from v$archived_log group by thread#;

THREAD# MAX(SEQUENCE#)
------- --------------
 1                 206

Standby database Details:

Oracle database version: 11.2.0.3 Enterprise Edition
Standby database name: DRPROC

Details with respect to the physical standby database:

SQL> select status,instance_name,database_role,open_mode from v$database,v$Instance;

STATUS       INSTANCE_NAME    DATABASE_ROLE    OPEN_MODE
------------ ---------------- ---------------- ----------------
OPEN         DRPROC           PHYSICAL STANDBY READ ONLY WITH APPLY

SQL> select thread#,max(sequence#) from v$archived_log where applied='YES' group by thread#;

THREAD# MAX(SEQUENCE#)
------- --------------
1                  206

SQL> select flashback_on from v$database;

FLASHBACK_ON
------------------
NO

You can observe that the standby database is in sync with the primary database. Below outcome shows that the Flash Recovery Area is configured on the physical standby database.
      
SQL> show parameter db_recovery_file_dest

NAME                         TYPE         VALUE
---------------------------  -----------  -------------
db_recovery_file_dest        string       +FRA_NEW
db_recovery_file_dest_size   big integer  4122M

Step 1: Cancel the Managed Recovery Process (MRP) on the physical standby database, shut it down and place it in Mount mode.

SQL> alter database recover managed standby database cancel;

Database altered.

SQL> shut immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup mount
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.
Total System Global Area 1269366784 bytes
Fixed Size                  2227984 bytes
Variable Size             805306608 bytes
Database Buffers          452984832 bytes
Redo Buffers                8847360 bytes
Database mounted.

Step 2: Once the standby database is mounted, convert the Physical standby database to snapshot standby database.

SQL> alter database convert to snapshot standby;

Database altered.

Step 3: You can now open the snapshot standby database and check its mode.
      
SQL> alter database open;

Database altered.

SQL> select status,instance_name,database_role,open_mode from v$database,v$Instance;

STATUS      INSTANCE_NAME   DATABASE_ROLE    OPEN_MODE
----------- --------------- ---------------- ------------------
OPEN        DRPROC            SNAPSHOT STANDBY READ WRITE

Small Test on the snapshot standby database.
1. Create a user called “SNAPTEST”
2. Create a table called “TEST” whose owner is “SNAPTEST” and insert some records in it. You can also update some of the records as well.
      
SQL> create user snaptest identified by oracle;

User created.

SQL> grant connect,resource to snaptest;

Grant succeeded.

SQL> conn snaptest/oracle@srps

Connected.

SQL> create table test(code number, name char(20));

Table created.

SQL> insert into test values (100,'ARUN');

1 row created.

SQL> insert into test values(200,'SHIVU');

1 row created.

SQL> commit;

Commit complete.

SQL> select * from test;

CODE       NAME
---------- --------------------
100        ARUN
200        SHIVU

SQL> update snaptest.test set code=500 where name='ARUN';

1 row updated.

SQL> commit;

Commit complete.

SQL> select * from snaptest.test;

CODE       NAME
---------- --------------------
500        ARUN
200        SHIVU

In the mean time, you can also see that the redo data from the primary database is received by the snapshot standby database but would not be applied.

On primary database the latest sequence generated is 208 and that on the standby database, the RFS process is idle for sequence 209.

Primary:
      
SQL> select thread#,max(sequence#) from v$archived_log group by thread#;

THREAD#    MAX(SEQUENCE#)
---------- --------------
 1         208

Standby:
      
SQL> select process,status,sequence# from v$managed_standby;

PROCESS   STATUS        SEQUENCE#
--------- ------------ ----------
ARCH      CLOSING               1
ARCH      CONNECTED             0
ARCH      CONNECTED             0
ARCH      CONNECTED             0
RFS       IDLE                  0
RFS       IDLE                209
RFS       IDLE                  0

7 rows selected.

Steps on converting back a snapshot standby database to physical standby database.
Step 1: Shut down the snapshot standby database and open it in Mount mode.

SQL> shut immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup mount
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 1269366784 bytes
Fixed Size                  2227984 bytes
Variable Size             805306608 bytes
Database Buffers          452984832 bytes
Redo Buffers                8847360 bytes
Database mounted.

Step 2: Convert the snapshot standby database to physical standby database.
      
SQL> alter database convert to physical standby;

Database altered.

Step 3: Once done, bounce the physical standby database and start the Managed Recovery Process (MRP) on it.
      
SQL> shut immediate
ORA-01507: database not mounted

ORACLE instance shut down.

SQL> startup
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 1269366784 bytes
Fixed Size                  2227984 bytes
Variable Size             805306608 bytes
Database Buffers          452984832 bytes
Redo Buffers                8847360 bytes
Database mounted.
Database opened.

SQL> select status,instance_name,database_role,open_mode from v$database,v$Instance;

STATUS       INSTANCE_NAME   DATABASE_ROLE     OPEN_MODE
------------ --------------  ----------------  ----------------
OPEN         DRPROC            PHYSICAL STANDBY  READ ONLY

SQL> alter database recover managed standby database disconnectfrom session;

Database altered.

SQL> select process,status,sequence# from v$managed_standby;

PROCESS   STATUS        SEQUENCE#
--------- ------------ ----------
ARCH      CONNECTED             0
ARCH      CONNECTED             0
ARCH      CONNECTED             0
ARCH      CONNECTED             0
RFS       IDLE                  0
RFS       IDLE                  0
RFS       IDLE                  0
MRP0      WAIT_FOR_LOG        213

8 rows selected.

Crosscheck whether the physical standby database is in sync with the primary database.

On Primary database:
      
SQL> select thread#,max(sequence#) from v$archived_log group bythread#;

THREAD#    MAX(SEQUENCE#)
---------- --------------
1          212

On Standby database:
      
SQL> select thread#,max(sequence#) from v$archived_log where applied='YES' group by thread#;

THREAD#    MAX(SEQUENCE#)
---------- --------------
1          212

You can see below that the transactions that were carried out when the standby database is opened in READ WRITE mode are flushed out after it was converted back to physical standby database mode.
      
SQL> select * from snaptest.test;

select * from snaptest.test
*
ERROR at line 1:
ORA-00942: table or view does not exist

SQL> select username,account_status from dba_users where username='SNAPTEST';

no rows selected

Drop/Rename Standby Redolog Files In Oracle


While performing the dataguard Broker, we need to drop the standby database while switchover the standby . As it seems an easy task but it is bit tricky . Below are the steps to drop the redolog file from standby database :

Dropping Standby Redolog Files:


On Standby Database : 
1. Check Redolog Files Status

SQL> select member,type from v$logfile;

GROUP#         MEMBER TYPE                                        
------         --------------------------------- --------
 1             /SAP/Oracle/OriglogA/redo_01a.log ONLINE   
 1             /SAP/Oracle/MirrlogA/redo_01b.log ONLINE     
 2             /SAP/Oracle/OriglogB/redo_02a.log STANDBY     
 2             /SAP/Oracle/MirrlogB/redo_02b.log STANDBY      3             /SAP/Oracle/OriglogC/redo_03a.log STANDBY      3             /SAP/Oracle/MirrlogC/redo_03b.log STANDBY     

2. We have to drop the standby redolog files .

SQL> alter database drop standby logfile group 1;
alter database drop standby logfile group 1
*
ERROR at line 1:
ORA-01156: recovery or flashback in progress may need access to files

NOTE: Now to solve this issue we have cancel the managed recovery session and set  "standby_file_management"  to manual and drop the standby redolog file  as 

SQL> alter database recover managed standby database cancel ;
Database altered.

SQL> alter system set standby_file_management='MANUAL' ;
System altered.

SQL>alter database drop standby logfile group 1;
Database altered.

SQL>alter database drop standby logfile group 2;
Database altered.

SQL>alter database drop standby logfile group 3;
Database altered.

If the status of standby redolog show the "clearing_current" then we cannot drop "clearing_current" status logs,and for that we have to sync with Primary and clear the log first before dropping as

SQL> alter database clear logfile group n;
Database altered.

Once the standby redologs are dropped then again back to recover the standby.


SQL>alter system set standby_file_management='AUTO' ;
System altered.

SQL> alter database recover managed standby database disconnect from session ;
Database altered.

Renaming Standby Redolog Files:

On Standby Database : 
1. Check Redolog Files Status

SQL> select member,type from v$logfile;

GROUP#         MEMBER TYPE                                        
------         --------------------------------- --------
 1             /SAP/Oracle/OriglogA/redo_01a.log ONLINE     
 1             /SAP/Oracle/MirrlogA/redo_01b.log ONLINE     
 2             /SAP/Oracle/OriglogB/redo_02a.log STANDBY     
 2             /SAP/Oracle/MirrlogB/redo_02b.log STANDBY      3             /SAP/Oracle/OriglogC/redo_03a.log STANDBY      3             /SAP/Oracle/MirrlogC/redo_03b.log STANDBY     

2. We have to drop the standby redolog files .

SQL> alter database drop standby logfile group 1;
alter database drop standby logfile group 1
*
ERROR at line 1:
ORA-01156: recovery or flashback in progress may need access to files

NOTE: Now to solve this issue we have cancel the managed recovery session and set  "standby_file_management"  to manual and drop the standby redolog file  as 

SQL> alter database recover managed standby database cancel ;
Database altered.

SQL> alter system set standby_file_management='MANUAL' ;
System altered.

SQL>alter database drop standby logfile group 1;
Database altered.

SQL> alter database clear logfile group 1;
Database altered.

SQL> alter database add standby logfile group 1 ('/SAP/ORADATA/OriglogA/redo_01a.log','/SAP/ORADATA/MirrlogA/redo_01b.log') size 500M reuse;

Database altered.

SQL>alter database drop standby logfile group 2;
Database altered.

SQL> alter database clear logfile group 2;
Database altered.

SQL> alter database add standby logfile group 2 ('/SAP/ORADATA/OriglogB/redo_02a.log','/SAP/ORADATA/MirrlogB/redo_02b.log') size 500M reuse;

Database altered.

SQL>alter database drop standby logfile group 3;
Database altered.

SQL> alter database clear logfile group 2;
Database altered.

SQL> alter database add standby logfile group 3 ('/SAP/ORADATA/OriglogC/redo_03a.log','/SAP/ORADATA/MirrlogC/redo_03b.log') size 500M reuse;

Database altered.

If the status of standby redolog show the "clearing_current" then we cannot drop "clearing_current" status logs,and for that we have to sync with Primary and clear the log first before dropping as



Once the standby redologs are dropped then again back to recover the standby.


SQL>alter system set standby_file_management='AUTO' ;
System altered.

SQL> alter database recover managed standby database disconnect from session ;
Database altered.

Finally check the status 

SQL> select member,type from v$logfile;

GROUP#         MEMBER TYPE                                        
------         --------------------------------- --------
 1             /SAP/ORADATA/OriglogA/redo_01a.log ONLINE   
 1             /SAP/ORADATA/MirrlogA/redo_01b.log ONLINE     
 2             /SAP/ORADATA/OriglogB/redo_02a.log STANDBY     
 2             /SAP/ORADATA/MirrlogB/redo_02b.log STANDBY      3             /SAP/ORADATA/OriglogC/redo_03a.log STANDBY      3             /SAP/ORADATA/MirrlogC/redo_03b.log STANDBY 

######################################################  All The Best ######################################################