RAC 11gR2模拟OCR和Votedisk损坏恢复过程

1）破坏前的ocr和votedisk信息检查检查ocr自动备份[root@rac1~]#ocrconfig-showbackuprac22013/10/1309:45:30/u01/grid/product/11.2.0/cdata/rac-cluster/backu

1）破坏前的ocr和votedisk信息检查

检查ocr自动备份

[root@rac1 ~]# ocrconfig -showbackup

rac2 2013/10/13 09:45:30 /u01/grid/product/11.2.0/cdata/rac-cluster/backup00.ocr

rac2 2013/10/13 05:45:29 /u01/grid/product/11.2.0/cdata/rac-cluster/backup01.ocr

rac2 2013/10/13 01:45:28 /u01/grid/product/11.2.0/cdata/rac-cluster/backup02.ocr

rac2 2013/10/12 01:45:26 /u01/grid/product/11.2.0/cdata/rac-cluster/day.ocr

rac2 2013/09/28 02:55:56 /u01/grid/product/11.2.0/cdata/rac-cluster/week.ocr

PROT-25: Manual backups for the Oracle Cluster Registry are not available

这里有一个PROT-25的提示信息，手工备份时无效，所以破坏后直接用自动备份恢复

检查ocr磁盘信息

[root@rac1 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

Version : 3

Total space (kbytes) : 262120

Used space (kbytes) : 2720

Available space (kbytes) : 259400

ID : 2026562699

Device/File Name : +OCRDATA

Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

查看votedisk磁盘信息

[root@rac1 ~]# crsctl query css votedisk

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 5c190e6ab4c04facbfdd4ca0e836a798 (ORCL:OCR1) [OCRDATA]

2. ONLINE abcc18afe6214fbcbfa02fad1c41b21b (ORCL:OCR2) [OCRDATA]

3. ONLINE 31e0a9df91514f73bf50a4e0a344af3d (ORCL:OCR3) [OCRDATA]

Located 3 voting disk(s).

可以看到OCR磁盘和votedisk都通过ASM OCRDATA磁盘组管理

查看ASM磁盘组 OCRDATA信息，我的OCRDATA磁盘组是有OCR1-3组成所以直接通过下列命令查询

[root@rac1 ~]# /etc/init.d/oracleasm querydisk -d OCR1

Disk "OCR1" is a valid ASM disk on device /dev/sda1[8,1]

[root@rac1 ~]# /etc/init.d/oracleasm querydisk -d OCR2

Disk "OCR2" is a valid ASM disk on device /dev/sdh1[8,113]

[root@rac1 ~]# /etc/init.d/oracleasm querydisk -d OCR3

Disk "OCR3" is a valid ASM disk on device /dev/sdb1[8,17]

由于ASM参数文件保存在OCRDATA下，所以对备份一份spfile

SQL> show parameter spfile;

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

spfile string +OCRDATA/rac-cluster/asmparame

terfile/registry.253.825083547

SQL> create pfile='/tmp/asmbak.ora' from spfile; --备份到/tmp/asmbak.ora

File created.

2）模拟损坏

使用dd命令破坏这几个磁盘,

[root@rac1 ~]# dd if=/dev/zero of=/dev/sda1 bs=1M count=10

10+0 records in

10+0 records out

10485760 bytes (10 MB) copied, 0.005454 seconds, 1.9 GB/s

[root@rac1 ~]# dd if=/dev/zero of=/dev/sdh1 bs=1M count=10

10+0 records in

10+0 records out

10485760 bytes (10 MB) copied, 0.00603 seconds, 1.7 GB/s

破坏后执行下面命令，发现各节点服务一切正常

[root@rac1 ~]# crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora.DATA.dg ora....up.type ONLINE ONLINE rac1

ora.FRA.dg ora....up.type ONLINE ONLINE rac1

ora....ER.lsnr ora....er.type ONLINE ONLINE rac1

ora....N1.lsnr ora....er.type ONLINE ONLINE rac1

ora.OCRDATA.dg ora....up.type ONLINE ONLINE rac1

ora.asm ora.asm.type ONLINE ONLINE rac1

ora.eons ora.eons.type ONLINE ONLINE rac1

ora.gsd ora.gsd.type OFFLINE OFFLINE

ora....network ora....rk.type ONLINE ONLINE rac1

ora.oc4j ora.oc4j.type OFFLINE OFFLINE

ora.ons ora.ons.type ONLINE ONLINE rac1

ora....SM1.asm application ONLINE ONLINE rac1

ora....C1.lsnr application ONLINE ONLINE rac1

ora.rac1.gsd application OFFLINE OFFLINE

ora.rac1.ons application ONLINE ONLINE rac1

ora.rac1.vip ora....t1.type ONLINE ONLINE rac1

ora....SM2.asm application ONLINE ONLINE rac2

ora....C2.lsnr application ONLINE ONLINE rac2

ora.rac2.gsd application OFFLINE OFFLINE

ora.rac2.ons application ONLINE ONLINE rac2

ora.rac2.vip ora....t1.type ONLINE ONLINE rac2

ora.ractest.db ora....se.type ONLINE ONLINE rac1

ora....ry.acfs ora....fs.type ONLINE ONLINE rac1

ora.scan1.vip ora....ip.type ONLINE ONLINE rac1

[root@rac1 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

Version : 3

Total space (kbytes) : 262120

Used space (kbytes) : 2720

Available space (kbytes) : 259400

ID : 2026562699

Device/File Name : +OCRDATA

Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

[root@rac1 ~]# crsctl query css votedisk

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 5c190e6ab4c04facbfdd4ca0e836a798 (ORCL:OCR1) [OCRDATA]

2. ONLINE abcc18afe6214fbcbfa02fad1c41b21b (ORCL:OCR2) [OCRDATA]

3. ONLINE 31e0a9df91514f73bf50a4e0a344af3d (ORCL:OCR3) [OCRDATA]

Located 3 voting disk(s).

停止RAC1crs服务

[root@rac1 ~]# crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'

CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'

CRS-2673: Attempting to stop 'ora.OCRDATA.dg' on 'rac1'

CRS-2673: Attempting to stop 'ora.ractest.db' on 'rac1'

CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac1'

CRS-2677: Stop of 'ora.registry.acfs' on 'rac1' succeeded

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'

CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2'

CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2'

CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeeded

CRS-2676: Start of 'ora.scan1.vip' on 'rac2' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac2'

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded

CRS-2677: Stop of 'ora.OCRDATA.dg' on 'rac1' succeeded

CRS-2677: Stop of 'ora.ractest.db' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac1'

CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rac1'

CRS-2677: Stop of 'ora.FRA.dg' on 'rac1' succeeded

CRS-2677: Stop of 'ora.DATA.dg' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'rac1'

CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'rac1'

CRS-2673: Attempting to stop 'ora.eons' on 'rac1'

CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'

CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded

CRS-2677: Stop of 'ora.eons' on 'rac1' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed

CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'

CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'

CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'

CRS-2673: Attempting to stop 'ora.asm' on 'rac1'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac1'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'

CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded

CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded

CRS-2677: Stop of 'ora.drivers.acfs' on 'rac1' succeeded

CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'

CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'

CRS-2673: Attempting to stop 'ora.diskmon' on 'rac1'

CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'

CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded

CRS-2677: Stop of 'ora.diskmon' on 'rac1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

3） 故障定位

在此启动发现已经无法正常启动了，跟踪alter日志，（由于系统日志没有发现什么重要信息，这里就没贴出来）

[root@rac1 ~]# tail -f /u01/grid/product/11.2.0/log/rac1/alertrac1.log

........

[ohasd(28327)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac1'.

2013-10-13 11:22:46.094

[cssd(28791)]CRS-1713:CSSD daemon is started in clustered mode

2013-10-13 11:22:46.178

[cssd(28791)]CRS-1637:Unable to locate configured voting file with ID 5c190e6a-b4c04fac-bfdd4ca0-e836a798; details at (:CSSNM00020:) in /u01/grid/product/11.2.0/log/rac1/cssd/ocssd.log

2013-10-13 11:22:46.179

[cssd(28791)]CRS-1637:Unable to locate configured voting file with ID abcc18af-e6214fbc-bfa02fad-1c41b21b; details at (:CSSNM00020:) in /u01/grid/product/11.2.0/log/rac1/cssd/ocssd.log

2013-10-13 11:22:46.179