1、備庫standby異常報錯

昨天凌晨磁碟空間突然暴漲導致oracle備庫異常，報警，後了過來清理掉磁碟的備份檔案，去了之後，歸檔日誌能同步過來了，但是啟動備庫standby發現mrp沒有啟動，後臺報錯如下，google了，說這種情況要重新再做備庫（不知道是否還有其它修復辦法呢？）：錯誤日誌報錯資訊如下：

Completed: alter database recover managed standby database using current logfile disconnect from session
Wed May 31 10:44:48 2017
Dumping diagnostic data in
 
 directory=[cdmp_20170531104448], requested by (instance=1, osid=43411 (PR00)), summary=[incident=600239].
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors with log /oracle/app/oracle/archivelogs/1_18799_940127349.dbf
MRP0: Background Media Recovery terminated with
 
 error 600
Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_pr00_43411.trc:
ORA-00600: internal error code, arguments: [2619], [18799], [], [], [], [], [], [], [], [], [], []
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_pr00_43411.trc:
ORA-00600
 
: internal error code, arguments: [2619], [18799], [], [], [], [], [], [], [], [], [], []
Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_mrp0_43408.trc  (incident=600231):
ORA-00600: internal error code, arguments: [2619], [18799], [], [], [], [], [], [], [], [], [], []
ORA-10877: error signaled in parallel recovery slave 
ORA-10877: error signaled in parallel recovery slave 
ORA-10877: error signaled in parallel recovery slave 
ORA-10877: error signaled in parallel recovery slave 
ORA-10877: error signaled in parallel recovery slave 

2、使用duplicate target 進行恢復備庫

duplicate target database for standby nofilenamecheck dorecover;

Starting Duplicate Db at 05-JUN-17
release channel c1;
release channel c2;
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=4708 device type=DISK

contents of Memory Script:
{
   set until scn  14430447592;
   restore clone standby controlfile;
}
executing Memory Script

executing command: SET until clause

Starting restore at 05-JUN-17
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: restoring control file
channel ORA_AUX_DISK_1: reading from backup piece /oracle/backup/data/controlfiles/c-3391761643-20170605-00
channel ORA_AUX_DISK_1: ORA-19870: error while restoring backup piece /oracle/backup/data/controlfiles/c-3391761643-20170605-00
ORA-19505: failed to identify file "/oracle/backup/data/controlfiles/c-3391761643-20170605-00"
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

這裡恢復失敗，google了下，沒有找到合適的辦法，咋辦呢？需要換種思路去恢復了，duplicate此路不通了。

3、嘗試使用最原始的rman備份直接restore、recover的方式

copy控制檔案、備份檔案，備份檔案已經有了，所以不需要再次生成，但是控制檔案需要再次生成一下，於是在主庫上生成最新的備份的控制檔案：

RMAN> backup current controlfile for standby format '/home/oracle/ctlfile4.bak';

Starting backup at 05-JUN-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=948 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including standby control file in backup set
channel ORA_DISK_1: starting piece 1 at 05-JUN-17
channel ORA_DISK_1: finished piece 1 at 05-JUN-17
piece handle=/home/oracle/ctlfile4.bak tag=TAG20170605T223329 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 05-JUN-17

Starting Control File and SPFILE Autobackup at 05-JUN-17
piece handle=/oracle/backup/data/controlfiles/c-3391761643-20170605-02 comment=NONE
Finished Control File and SPFILE Autobackup at 05-JUN-17

RMAN> 

scp將控制檔案和備份檔案copy到備庫上去

scp -r 2017-06-05 101.251.31.131:/oracle/
scp /home/oracle/ctlfile4.bak 101.251.31.131:/home/oracle/

接下來基本全在備庫上進行操作，進行restore standby controlfile恢復控制檔案

RMAN> restore standby controlfile from '/home/oracle/ctlfile4.bak';

Starting restore at 05-JUN-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=4708 device type=DISK

channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
output file name=/oracle/app/oracle/oradata/powerdes/control01.ctl
output file name=/oracle/app/oracle/fast_recovery_area/powerdes/control02.ctl
Finished restore at 05-JUN-17

RMAN> 

啟動到mount狀態

RMAN> alter database mount;

database mounted
released channel: ORA_DISK_1

RMAN

註冊備份集合

RMAN> catalog start with '/oracle/2017-06-05';

Starting implicit crosscheck backup at 05-JUN-17
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=4709 device type=DISK
Crosschecked 147 objects
Finished implicit crosscheck backup at 05-JUN-17

Starting implicit crosscheck copy at 05-JUN-17
using channel ORA_DISK_1
Crosschecked 2 objects
Finished implicit crosscheck copy at 05-JUN-17

searching for all files in the recovery area
cataloging files...
no files cataloged

searching for all files that match the pattern /oracle/2017-06-05

List of Files Unknown to the Database
=====================================
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9901.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9899.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9910.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9912.bak
File Name: /oracle/2017-06-05/rman_backup.log
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9908.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9904.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9905.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9906.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9909.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9903.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9902.bak
File Name: /oracle/2017-06-05/full_POWERDES_20170605_9911.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9900.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9907.bak

Do you really want to catalog the above files (enter YES or NO)? YES
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9901.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9899.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9910.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9912.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9908.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9904.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9905.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9906.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9909.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9903.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9902.bak
File Name: /oracle/2017-06-05/full_POWERDES_20170605_9911.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9900.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9907.bak

List of Files Which Where Not Cataloged
=======================================
File Name: /oracle/2017-06-05/rman_backup.log
  RMAN-07517: Reason: The file header is corrupted

RMAN>

RMAN> restore database;

Starting restore at 05-JUN-17
using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /home/oradata/powerdes/system01.dbf
channel ORA_DISK_1: restoring datafile 00002 to /home/oradata/powerdes/sysaux01.dbf
channel ORA_DISK_1: restoring datafile 00003 to /home/oradata/powerdes/undotbs01.dbf
channel ORA_DISK_1: restoring datafile 00004 to /home/oradata/powerdes/users01.dbf
channel ORA_DISK_1: restoring datafile 00005 to /home/oradata/powerdes/powerdesk01.dbf
channel ORA_DISK_1: restoring datafile 00006 to /home/oradata/powerdes/plas01.dbf
channel ORA_DISK_1: restoring datafile 00007 to /home/oradata/powerdes/pl01.dbf
channel ORA_DISK_1: restoring datafile 00008 to /home/oradata/powerdes/help01.dbf
channel ORA_DISK_1: restoring datafile 00009 to /home/oradata/powerdes/adobelc01.dbf
channel ORA_DISK_1: restoring datafile 00010 to /home/oradata/powerdes/sms01.dbf
channel ORA_DISK_1: restoring datafile 00011 to /home/oradata/powerdes/plcrm01.dbf
channel ORA_DISK_1: restoring datafile 00012 to /home/oradata/powerdes/powerdesk02.dbf
channel ORA_DISK_1: restoring datafile 00013 to /home/oradata/powerdes/datagm01.dbf
channel ORA_DISK_1: restoring datafile 00014 to /home/oradata/powerdes/plimp01.DBF
channel ORA_DISK_1: restoring datafile 00015 to /home/oradata/powerdes/dwetl01.DBF
channel ORA_DISK_1: restoring datafile 00016 to /home/oradata/powerdes/dw02.DBF
channel ORA_DISK_1: restoring datafile 00017 to /home/oradata/powerdes/timdba01.DBF
channel ORA_DISK_1: restoring datafile 00018 to /home/oradata/powerdes/users02.dbf
channel ORA_DISK_1: restoring datafile 00019 to /home/oradata/powerdes/system02.dbf
channel ORA_DISK_1: restoring datafile 00020 to /home/oradata/powerdes/powerdesk03.dbf
channel ORA_DISK_1: restoring datafile 00021 to /home/oradata/powerdes/timdba02.dbf
channel ORA_DISK_1: reading from backup piece /oracle/2017-06-05/full_POWERDES_20170605_9911.bak

channel ORA_DISK_1: piece handle=/oracle/2017-06-05/full_POWERDES_20170605_9911.bak tag=TAG20170605T033303
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 01:29:37
Finished restore at 05-JUN-17

RMAN> 

開始rocover database操作

RMAN> recover database;

Starting recover at 05-JUN-17
using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 20421 is already on disk as file /oracle/app/oracle/archivelogs/1_20421_940127349.dbf
archived log for thread 1 with sequence 20422 is already on disk as file /oracle/app/oracle/archivelogs/1_20422_940127349.dbf
archived log for thread 1 with sequence 20423 is already on disk as file /oracle/app/oracle/archivelogs/1_20423_940127349.dbf
archived log for thread 1 with sequence 20424 is already on disk as file /oracle/app/oracle/archivelogs/1_20424_940127349.dbf
archived log for thread 1 with sequence 20425 is already on disk as file /oracle/app/oracle/archivelogs/1_20425_940127349.dbf
archived log for thread 1 with sequence 20426 is already on disk as file /oracle/app/oracle/archivelogs/1_20426_940127349.dbf
archived log for thread 1 with sequence 20427 is already on disk as file /oracle/app/oracle/archivelogs/1_20427_940127349.dbf
archived log for thread 1 with sequence 20428 is already on disk as file /oracle/app/oracle/archivelogs/1_20428_940127349.dbf
archived log for thread 1 with sequence 20429 is already on disk as file /oracle/app/oracle/archivelogs/1_20429_940127349.dbf
archived log for thread 1 with sequence 20430 is already on disk as file /oracle/app/oracle/archivelogs/1_20430_940127349.dbf
archived log for thread 1 with sequence 20431 is already on disk as file /oracle/app/oracle/archivelogs/1_20431_940127349.dbf
archived log for thread 1 with sequence 20432 is already on disk as file /oracle/app/oracle/archivelogs/1_20432_940127349.dbf
archived log for thread 1 with sequence 20433 is already on disk as file /oracle/app/oracle/archivelogs/1_20433_940127349.dbf
archived log for thread 1 with sequence 20434 is already on disk as file /oracle/app/oracle/archivelogs/1_20434_940127349.dbf
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20357
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20358
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20359
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20360
channel ORA_DISK_1: reading from backup piece /oracle/2017-06-05/arch_POWERDES_20170605_9912.bak
channel ORA_DISK_1: piece handle=/oracle/2017-06-05/arch_POWERDES_20170605_9912.bak tag=TAG20170605T051030
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:15
archived log file name=/oracle/app/oracle/archivelogs/1_20357_940127349.dbf thread=1 sequence=20357
archived log file name=/oracle/app/oracle/archivelogs/1_20358_940127349.dbf thread=1 sequence=20358
archived log file name=/oracle/app/oracle/archivelogs/1_20359_940127349.dbf thread=1 sequence=20359
archived log file name=/oracle/app/oracle/archivelogs/1_20360_940127349.dbf thread=1 sequence=20360
unable to find archived log
archived log thread=1 sequence=20361
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 06/05/2017 22:17:07
RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 20361 and starting SCN of 14425991740

RMAN>  recover database until scn 14425991740;

Starting recover at 05-JUN-17
using channel ORA_DISK_1

starting media recovery
Oracle Error: 
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup 
ORA-01110: data file 1: '/home/oradata/powerdes/system01.dbf'

media recovery complete, elapsed time: 00:00:01

Finished recover at 05-JUN-17

RMAN>

最後開始啟動傳輸並且應用歸檔日誌

SQL> alter database recover managed standby database using current logfile disconnect from session;

Database altered.

SQL> 

4、開始檢查主庫備庫資料一致性

檢視歸檔日誌應用情況：

SQL> select sequence#,applied from v$archived_log order by sequence# asc;

 SEQUENCE# APPLIED
---------- ---------
     20379 YES
     20380 YES
     20381 YES
     20382 YES
     20383 YES
     20384 YES
     20385 YES
     20386 YES
     20387 YES
     20388 YES
     20389 YES

 SEQUENCE# APPLIED
---------- ---------
     20390 YES
     20391 YES
     20392 YES
     20393 YES
     20394 YES
     20395 YES
     20396 YES
     20397 YES
     20398 NO
     20399 NO
     20400 NO

檢視後臺alert日誌，正在進行應用

Media Recovery Log /oracle/app/oracle/archivelogs/1_20391_940127349.dbf
Mon Jun 05 22:24:00 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20392_940127349.dbf
Mon Jun 05 22:24:10 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20393_940127349.dbf
Mon Jun 05 22:24:22 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20394_940127349.dbf
Mon Jun 05 22:24:33 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20395_940127349.dbf
Mon Jun 05 22:24:47 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20396_940127349.dbf
Mon Jun 05 22:25:00 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20397_940127349.dbf

檢視備庫主庫log資訊是保持一致的，如下所示：

SQL> archive log list;
Database log mode          Archive Mode
Automatic archival         Enabled
Archive destination        /oracle/app/oracle/archivelogs
Oldest online log sequence     20430
Next log sequence to archive   20435
Current log sequence           20435
SQL> 
SQL> archive log list;
Database log mode          Archive Mode
Automatic archival         Enabled
Archive destination        /oracle/app/oracle/archivelogs
Oldest online log sequence     20430
Next log sequence to archive   0
Current log sequence           20435
SQL>

至此，利用rman備份直接restore、recover的方式恢復了dataguard的standby庫。


仔細思考下，這個問題以前一直沒有遇到過，磁碟滿了後，按照道理來說，清理磁碟後，可以直接恢復得，但是卻不能恢復，原因猜測在於啟用了實時應用歸檔日誌導致。這樣就導致了重新再次傳輸載入的時候，混亂了。去檢視以前的dataguard搭建記錄http://blog.csdn.net/mchdba/article/details/71036947，發現有類似的操作記錄：

（2）新增current logfile引數，使得應用當前正在讀寫，還沒有完成歸檔的日誌
SQL> alter database recover managedstandby database using current logfile disconnect from session;