Oracle 11gR2 RAC重启后只能起单节点

将Oracle 11gR2 RAC正常部署完成之后执行两节点重启操作发现其中有一个节点的集群资源无法启动，遂再次重启该无法启动集群资源的

问题背景：

将Oracle 11gR2 RAC正常部署完成之后执行两节点重启操作发现其中有一个节点的集群资源无法启动，遂再次重启该无法启动集群资源的节点，，还是不可。随即将正常节点重启发现原故障节点资源起来了，待重启完毕后原正常节点资源无法启动。

集群环境：

OS:RedHat EnterPrise5.8 x86_x64

DB:Oracle EnterPrise Database 11.2.0.4.0 x86_x64

GRID:Oracle Grid Infrastructure 11.2.0.4 x86_x64

心跳和公网网卡均做bond1绑定；

存储是IBM的，故采用RDAC做多路径聚合;

问题分析:

检查OS系统日志和集群日志

Oracle 11g 在RedHat Linux 5.8_x64平台的安装手册

Linux-6-64下安装Oracle 12C笔记

在CentOS 6.4下安装Oracle 11gR2(x64)

Oracle 11gR2 在VMWare虚拟机中安装步骤

Debian 下 安装 Oracle 11g XE R2

db01 grid 日志信息

2014-09-29 15:36:36.587:
[ctssd(12616)]CRS-2405:The Cluster Time Synchronization Service on host db01 is shutdown by user
2014-09-29 15:36:36.589:
[mdnsd(6173)]CRS-5602:mDNS service stopping by request.
[client(17411)]CRS-10001:29-Sep-14 15:36 ACFS-9290: Waiting for ASM to shutdown.
2014-09-29 15:36:46.395:
[cssd(8958)]CRS-1603:CSSD on node db01 shutdown by user.
2014-09-29 15:36:46.509:
[ohasd(12463)]CRS-2767:Resource state recovery not attempted for 'ora.cssdmonitor' as its target state is OFFLINE
2014-09-29 15:36:46.509:
[ohasd(12463)]CRS-2769:Unable to failover resource 'ora.cssdmonitor'.
2014-09-29 15:36:46.608:
[cssd(8958)]CRS-1660:The CSS daemon shutdown has completed

ocssd
2014-09-29 15:36:46.052: [    CSSD][1122224448]clssnmSendingThread: sending status msg to all nodes
2014-09-29 15:36:46.053: [    CSSD][1122224448]clssnmSendingThread: sent 4 status msgs to all nodes
2014-09-29 15:36:46.388: [    CSSD][1089472832]clssgmExitGrock: client 1 (0x1261ee30), grock haip.cluster_interconnect, member 0
2014-09-29 15:36:46.388: [    CSSD][1089472832]clssgmUnregisterPrimary: Unregistering member 0 (0x1261ace0) in global grock haip.cluster_interconnect
2014-09-29 15:36:46.388: [    CSSD][1089472832]clssgmUnreferenceMember: global grock haip.cluster_interconnect member 0 refcount is 0
2014-09-29 15:36:46.388: [    CSSD][1089472832]clssgmAllocateRPCIndex: allocated rpc 357 (0x2aaaaaee8a58)
2014-09-29 15:36:46.388: [    CSSD][1089472832]clssgmRPC: rpc 0x2aaaaaee8a58 (RPC#357) tag(165002a) sent to node 1
2014-09-29 15:36:46.388: [    CSSD][1079531840]clssgmHandleMasterMemberExit: [s(1) d(1)]
2014-09-29 15:36:46.388: [    CSSD][1079531840]clssgmRemoveMember: grock haip.cluster_interconnect, member number 0 (0x1261ace0) node number 1 state 0x4 grock type 2
2014-09-29 15:36:46.388: [    CSSD][1079531840]clssgmGrantLocks: 0-> new master (1/2) group haip.cluster_interconnect
2014-09-29 15:36:46.388: [    CSSD][1079531840]clssgmRPCDone: rpc 0x2aaaaaee8a58 (RPC#357) state 6, flags 0x100
2014-09-29 15:36:46.388: [    CSSD][1079531840]clssgmDelMemCmpl: rpc 0x2aaaaaee8a58, ret 0, client 0x1261ee30 member 0x1261ace0
2014-09-29 15:36:46.388: [    CSSD][1079531840]clssgmFreeRPCIndex: freeing rpc 357
2014-09-29 15:36:46.388: [    CSSD][1079531840]clssgmAllocateRPCIndex: allocated rpc 358 (0x2aaaaaee8b00)
2014-09-29 15:36:46.388: [    CSSD][1089472832]clssgmDiscEndpcl: gipcDestroy 0x1a370
2014-09-29 15:36:46.388: [    CSSD][1079531840]clssgmRPCBroadcast: rpc(0x166002a), status(1), sendcount(1), filtered by specific properties:
2014-09-29 15:36:46.388: [    CSSD][1089472832]clssgmDeadProc: proc 0x1262a8a0
2014-09-29 15:36:46.388: [    CSSD][1089472832]clssgmDestroyProc: cleaning up proc(0x1262a8a0) con(0x1a341) skgpid  ospid 12515 with 0 clients, refcount 0
2014-09-29 15:36:46.388: [    CSSD][1089472832]clssgmDiscEndpcl: gipcDestroy 0x1a341
2014-09-29 15:36:46.389: [    CSSD][1079531840]clssgmRPCDone: rpc 0x2aaaaaee8b00 (RPC#358) state 4, flags 0x402
2014-09-29 15:36:46.389: [    CSSD][1079531840]clssgmBroadcastGrockRcfgCmpl: RPC(0x166002a) of grock(haip.cluster_interconnect) received all acks, grock update sequence(4)
2014-09-29 15:36:46.389: [    CSSD][1079531840]clssgmFreeRPCIndex: freeing rpc 358
2014-09-29 15:36:46.395: [    CSSD][1089472832]clssgmExecuteClientRequest: MAINT recvd from proc 7 (0x1264c8e0)
2014-09-29 15:36:46.395: [    CSSD][1089472832]clssgmShutDown: Received explicit shutdown request from client.
2014-09-29 15:36:46.395: [    CSSD][1089472832]clssgmClientShutdown: total iocapables 0
2014-09-29 15:36:46.395: [    CSSD][1089472832]clssgmClientShutdown: graceful shutdown completed.
2014-09-29 15:36:46.395: [    CSSD][1089472832]clssgmClientShutdown: signaling to the agent that resource should remain down
2014-09-29 15:36:46.395: [    CSSD][1089472832]clssgmCompareSwapEventValue: changed CmInfo State  val 0, from 11, changes 21
2014-09-29 15:36:46.395: [    CSSD][1079531840]clssgmPeerListener: terminating at incarn(307480783)
2014-09-29 15:36:46.395: [    CSSD][1079531840]clssgmPeerDeactivate: node 2 (db02), death 0, state 0x1 connstate 0xf
2014-09-29 15:36:46.395: [    CSSD][1079531840]clssgmCleanFuture: discarded 0 future msgs for 2
2014-09-29 15:36:46.395: [    CSSD][1079531840]clssgmDiscEndppl: gipcDestroy 0x26027
2014-09-29 15:36:46.496: [    CSSD][1089472832]clssnmSendManualShut: Notifying all nodes that this node has been manually shut down
2014-09-29 15:36:46.497: [GIPCHAUP][1091049792] gipchaUpperDisconnect: initiated discconnect umsg 0x12ac30d0 { msg 0x12ad9f08, ret gipcretRequestPending (15), flags 0x2 }, msg 0x12ad9f08 { type gipchaMsgTypeDisconnect (5), srcCid 00000000-00026038, dstCid 00000000-00000d5b }, endp 0x12ac12d0 [0000000000026038] { gipchaEndpoint : port '5996-da20-cc7e-b119', peer 'db02:gm2_db-cluster/2991-10ef-6fca-054c', srcCid 00000000-00026038,  dstCid 00000000-00000d5b, numSend 0, maxSend 100, groupListType 2, hagroup 0x12a9f4e0, usrFlags 0x4000, flags 0x21c }
2014-09-29 15:36:46.497: [GIPCHAUP][1091049792] gipchaUpperCallbackDisconnect: completed DISCONNECT ret gipcretSuccess (0), umsg 0x12ac30d0 { msg 0x12ad9f08, ret gipcretSuccess (0), flags 0x2 }, msg 0x12ad9f08 { type gipchaMsgTypeDisconnect (5), srcCid 00000000-00026038, dstCid 00000000-00000d5b }, hendp 0x12ac12d0 [0000000000026038] { gipchaEndpoint : port '5996-da20-cc7e-b119', peer 'db02:gm2_db-cluster/2991-10ef-6fca-054c', srcCid 00000000-00026038,  dstCid 00000000-00000d5b, numSend 0, maxSend 100, groupListType 2, hagroup 0x12a9f4e0, usrFlags 0x4000, flags 0x21c }
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: Aborting client (0x2aaaac8b3180) proc (0x2aaaac937fd0), iocapables 1.
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: I/O capable proc (0x2aaaac937fd0), pid (6300), iocapables 1, client (0x2aaaac8b3180)
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: Aborting client (0x2aaaac9773e0) proc (0x2aaaac937fd0), iocapables 2.
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: I/O capable proc (0x2aaaac937fd0), pid (6300), iocapables 2, client (0x2aaaac9773e0)
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: Aborting client (0x2aaaac958d50) proc (0x2aaaac99d570), iocapables 3.
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: I/O capable proc (0x2aaaac99d570), pid (6208), iocapables 3, client (0x2aaaac958d50)
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: Aborting client (0x12645650) proc (0x1264c8e0), iocapables 4.
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: I/O capable proc (0x1264c8e0), pid (12518), iocapables 4, client (0x12645650)
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: Aborting client (0x126171b0) proc (0x1264c8e0), iocapables 5.
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: I/O capable proc (0x1264c8e0), pid (12518), iocapables 5, client (0x126171b0)
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: Aborting client (0x1264cef0) proc (0x1264c8e0), iocapables 6.
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: I/O capable proc (0x1264c8e0), pid (12518), iocapables 6, client (0x1264cef0)
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: Aborting client (0x125f0340) proc (0x1260f910), iocapables 7.
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: I/O capable proc (0x1260f910), pid (12520), iocapables 7, client (0x125f0340)
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: Aborting client (0x125e98b0) proc (0x1260f910), iocapables 8.
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: I/O capable proc (0x1260f910), pid (12520), iocapables 8, client (0x125e98b0)
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: Aborting client (0x12ac2790) proc (0x1260f910), iocapables 9.
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmSendShutdown: I/O capable proc (0x1260f910), pid (12520), iocapables 9, client (0x12ac2790)
2014-09-29 15:36:46.506: [    CSSD][1089472832]clssgmClientShutdown: sending shutdown, fence_done 1
2014-09-29 15:36:46.608: [ default][1089472832]kgzf_fini: called