OEL6.3上 Oracle RAC 上节点驱逐检查过程

RAC上是vmware vspere的虚拟机上有两个OEL6.3的虚拟机，上面跑的库是11.2.0.4

RAC上是vmware vspere的虚拟机上有两个OEL6.3的虚拟机，，上面跑的库是11.2.0.4

发现节点2挂掉了

1.检查节点1的alert
 [root@racnode1 racnode1]# pwd
 /u01/apps/grid/gridhome/11.2.0/grid/log/racnode1
 [root@racnode1 racnode1]# tail -1000 alertracnode1.log
 2014-04-15 09:41:20.815:
 [crsd(27311)]CRS-2765:Resource 'ora.net1.network' has failed on server 'racnode1'.
 2014-04-15 09:41:42.760:
 [cssd(26972)]CRS-1612:Network communication with node racnode2 (2) missing for 50% of timeout interval.  Removal of this node from cluster in 15.000 seconds
 2014-04-15 09:41:50.763:
 [cssd(26972)]CRS-1611:Network communication with node racnode2 (2) missing for 75% of timeout interval.  Removal of this node from cluster in 7.000 seconds
 2014-04-15 09:41:54.764:
 [cssd(26972)]CRS-1610:Network communication with node racnode2 (2) missing for 90% of timeout interval.  Removal of this node from cluster in 3.000 seconds
 2014-04-15 09:41:57.766:
 [cssd(26972)]CRS-1607:Node racnode2 is being evicted in cluster incarnation 291818318; details at (:CSSNM00007:) in /u01/apps/grid/gridhome/11.2.0/grid/log/racnode1/cssd/ocssd.log.
 2014-04-15 09:42:06.052:
 [cssd(26972)]CRS-1625:Node racnode2, number 2, was manually shut down
 2014-04-15 09:42:06.059:
 [cssd(26972)]CRS-1601:CSSD Reconfiguration complete. Active nodes are racnode1 .
 2014-04-15 09:42:06.950:
 [crsd(27311)]CRS-5504:Node down event reported for node 'racnode2'.
 2014-04-15 09:42:24.882:
 [crsd(27311)]CRS-2773:Server 'racnode2' has been removed from pool 'Generic'.
 2014-04-15 09:42:24.882:
 [crsd(27311)]CRS-2773:Server 'racnode2' has been removed from pool 'ora.pera'.
 [root@racnode1 racnode1]#
 

节点2被驱逐时间：2014-04-15 09:41:57
 

2.查看ocssd.log
 # more /u01/apps/grid/gridhome/11.2.0/grid/log/racnode1/cssd/ocssd.log |grep "2014-04-15 09:41"
 

2014-04-15 09:41:23.620: [    CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
 2014-04-15 09:41:23.621: [    CSSD][906479360]clssnmSendingThread: sent 4 status msgs to all nodes
 2014-04-15 09:41:28.758: [    CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
 2014-04-15 09:41:28.758: [    CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
 2014-04-15 09:41:33.145: [GIPCHGEN][919283456] gipchaInterfaceFail: marking interface failing 0x7fc01821d4c0 { host '', haName 'CSS_racnode-cluster', local (nil), ip '172.168.1.11:52955', subnet '172.168.1.0', mask '255.255.255.0', mac '00-50-56-a1-7b-e2', ifname 'eth1', numRef 1, numFail 0, idxBoot 0, flags 0x184d }
 2014-04-15 09:41:33.760: [    CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
 2014-04-15 09:41:33.760: [GIPCHGEN][920860416] gipchaInterfaceFail: marking interface failing 0x7fc024040a20 { host 'racnode2', haName 'CSS_racnode-cluster', local 0x7fc01821d4c0, ip '172.168.1.12:60678', subnet '172.168.1.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 4, flags 0x6 }
 2014-04-15 09:41:33.760: [    CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
 2014-04-15 09:41:33.760: [GIPCHGEN][920860416] gipchaInterfaceDisable: disabling interface 0x7fc01821d4c0 { host '', haName 'CSS_racnode-cluster', local (nil), ip '172.168.1.11:52955', subnet '172.168.1.0', mask '255.255.255.0', mac '00-50-56-a1-7b-e2', ifname 'eth1', numRef 0, numFail 1, idxBoot 0, flags 0x19cd }
 2014-04-15 09:41:33.760: [GIPCHGEN][920860416] gipchaInterfaceDisable: disabling interface 0x7fc024040a20 { host 'racnode2', haName 'CSS_racnode-cluster', local 0x7fc01821d4c0, ip '172.168.1.12:60678', subnet '172.168.1.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 4, flags 0x86 }
 2014-04-15 09:41:33.761: [GIPCHALO][920860416] gipchaLowerCleanInterfaces: performing cleanup of disabled interface 0x7fc024040a20 { host 'racnode2', haName 'CSS_racnode-cluster', local 0x7fc01821d4c0, ip '172.168.1.12:60678', subnet '172.168.1.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 4, flags 0xa6 }
 2014-04-15 09:41:33.885: [GIPCHDEM][920860416] gipchaWorkerCleanInterface: performing cleanup of disabled interface 0x7fc01821d4c0 { host '', haName 'CSS_racnode-cluster', local (nil), ip '172.168.1.11:52955', subnet '172.168.1.0', mask '255.255.255.0', mac '00-50-56-a1-7b-e2', ifname 'eth1', numRef 0, numFail 0, idxBoot 0, flags 0x19ed }
 2014-04-15 09:41:34.145: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest,  hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
 2014-04-15 09:41:37.993: [GIPCHALO][920860416] gipchaLowerProcessNode: no valid interfaces found to node for 5240 ms, node 0x7fc018226540 { host 'racnode2', haName 'CSS_racnode-cluster', srcLuid 8bbba732-d4368a71, dstLuid 0fc9914c-ec29f220 numInf 0, contigSeq 1787047, lastAck 1786995, lastValidAck 1787046, sendSeq [1786996 : 1787014], createTime 1683284, sentRegister 1, localMonitor 1, flags 0x2408 }
 2014-04-15 09:41:38.762: [    CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
 2014-04-15 09:41:38.762: [    CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
 2014-04-15 09:41:40.146: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest,  hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
 2014-04-15 09:41:42.760: [    CSSD][908056320]clssnmPollingThread: node racnode2 (2) at 50% heartbeat fatal, removal in 15.000 seconds
 2014-04-15 09:41:42.760: [    CSSD][908056320]clssnmPollingThread: node racnode2 (2) is impending reconfig, flag 2294796, misstime 15000
 2014-04-15 09:41:42.760: [    CSSD][908056320]clssnmPollingThread: local diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig status(1)
 2014-04-15 09:41:42.764: [    CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
 2014-04-15 09:41:42.764: [    CSSD][906479360]clssnmSendingThread: sent 4 status msgs to all nodes
 2014-04-15 09:41:43.764: [GIPCHALO][920860416] gipchaLowerProcessNode: no valid interfaces found to node for 11010 ms, node 0x7fc018226540 { host 'racnode2', haName 'CSS_racnode-cluster', srcLuid 8bbba732-d4368a71, dstLuid 0fc9914c-ec29f220 numInf 0, contigSeq 1787047, lastAck 1786995, lastValidAck 1787046, sendSeq [1786996 : 1787025], createTime 1683284, sentRegister 1, localMonitor 1, flags 0x2408 }
 2014-04-15 09:41:43.822: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526103/1203750934
 2014-04-15 09:41:43.906: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1193997, LATS 1203751024, lastSeqNo 1175771, uniqueness 1396323841, timestamp 1397526102/1203310914
 2014-04-15 09:41:45.102: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526105/1203752214
 2014-04-15 09:41:45.772: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526105/1203752884
 2014-04-15 09:41:46.147: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest,  hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
 2014-04-15 09:41:46.181: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1193999, LATS 1203753294, lastSeqNo 1193997, uniqueness 1396323841, timestamp 1397526104/1203313164
 2014-04-15 09:41:46.782: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526106/1203753894
 2014-04-15 09:41:47.120: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194001, LATS 1203754234, lastSeqNo 1193999, uniqueness 1396323841, timestamp 1397526106/1203314834
 2014-04-15 09:41:47.622: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526107/1203754734
 2014-04-15 09:41:47.765: [    CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
 2014-04-15 09:41:47.765: [    CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
 2014-04-15 09:41:48.004: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194002, LATS 1203755124, lastSeqNo 1194001, uniqueness 1396323841, timestamp 1397526107/1203315684
 2014-04-15 09:41:48.507: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526108/1203755624
 2014-04-15 09:41:48.774: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194003, LATS 1203755894, lastSeqNo 1194002, uniqueness 1396323841, timestamp 1397526108/1203316514
 2014-04-15 09:41:49.213: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526109/1203756324
 2014-04-15 09:41:49.766: [GIPCHALO][920860416] gipchaLowerProcessNode: no valid interfaces found to node for 17010 ms, node 0x7fc018226540 { host 'racnode2', haName 'CSS_racnode-cluster', srcLuid 8bbba732-d4368a71, dstLuid 0fc9914c-ec29f220 numInf 0, contigSeq 1787047, lastAck 1786995, lastValidAck 1787046, sendSeq [1786996 : 1787037], createTime 1683284, sentRegister 1, localMonitor 1, flags 0x2408 }
 2014-04-15 09:41:49.893: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194004, LATS 1203757004, lastSeqNo 1194003, uniqueness 1396323841, timestamp 1397526109/1203317334
 2014-04-15 09:41:49.903: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526109/1203757014
 2014-04-15 09:41:50.528: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194005, LATS 1203757644, lastSeqNo 1194004, uniqueness 1396323841, timestamp 1397526109/1203317964
 2014-04-15 09:41:50.548: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526110/1203757664
 2014-04-15 09:41:50.763: [    CSSD][908056320]clssnmPollingThread: node racnode2 (2) at 75% heartbeat fatal, removal in 7.000 seconds
 2014-04-15 09:41:50.875: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194006, LATS 1203757994, lastSeqNo 1194005, uniqueness 1396323841, timestamp 1397526110/1203318664
 2014-04-15 09:41:51.113: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526111/1203758224
 2014-04-15 09:41:51.396: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194007, LATS 1203758514, lastSeqNo 1194006, uniqueness 1396323841, timestamp 1397526110/1203319204
 2014-04-15 09:41:51.692: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526111/1203758804
 2014-04-15 09:41:52.148: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest,  hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
 2014-04-15 09:41:52.294: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194008, LATS 1203759404, lastSeqNo 1194007, uniqueness 1396323841, timestamp 1397526111/1203319754
 2014-04-15 09:41:52.305: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526112/1203759424
 2014-04-15 09:41:52.767: [    CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
 2014-04-15 09:41:52.767: [    CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
 2014-04-15 09:41:52.772: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194009, LATS 1203759884, lastSeqNo 1194008, uniqueness 1396323841, timestamp 1397526112/1203320364
 2014-04-15 09:41:52.974: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526112/1203760084
 2014-04-15 09:41:53.793: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194010, LATS 1203760904, lastSeqNo 1194009, uniqueness 1396323841, timestamp 1397526112/1203321034
 2014-04-15 09:41:53.934: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526113/1203761044
 2014-04-15 09:41:54.401: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194012, LATS 1203761514, lastSeqNo 1194010, uniqueness 1396323841, timestamp 1397526114/1203322404
 2014-04-15 09:41:54.494: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526114/1203761604
 2014-04-15 09:41:54.764: [    CSSD][908056320]clssnmPollingThread: node racnode2 (2) at 90% heartbeat fatal, removal in 3.000 seconds, seedhbimpd 1
 2014-04-15 09:41:55.044: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526115/1203762154
 2014-04-15 09:41:55.134: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194013, LATS 1203762254, lastSeqNo 1194012, uniqueness 1396323841, timestamp 1397526114/1203322964
 2014-04-15 09:41:55.640: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526115/1203762754
 2014-04-15 09:41:55.768: [GIPCHALO][920860416] gipchaLowerProcessNode: no valid interfaces found to node for 23010 ms, node 0x7fc018226540 { host 'racnode2', haName 'CSS_racnode-cluster', srcLuid 8bbba732-d4368a71, dstLuid 0fc9914c-ec29f220 numInf 0, contigSeq 1787047, lastAck 1786995, lastValidAck 1787046, sendSeq [1786996 : 1787048], createTime 1683284, sentRegister 1, localMonitor 1, flags 0x2408 }
 2014-04-15 09:41:56.498: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526116/1203763614
 2014-04-15 09:41:57.145: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194015, LATS 1203764254, lastSeqNo 1194013, uniqueness 1396323841, timestamp 1397526116/1203324554
 2014-04-15 09:41:57.645: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526117/1203764764
 2014-04-15 09:41:57.764: [    CSSD][908056320]clssnmPollingThread: Removal started for node racnode2 (2), flags 0x23040c, state 3, wt4c 0
 2014-04-15 09:41:57.764: [    CSSD][908056320]clssnmMarkNodeForRemoval: node 2, racnode2 marked for removal
 2014-04-15 09:41:57.765: [    CSSD][908056320]clssnmDiscHelper: racnode2, node(2) connection failed, endp (0x3938), probe(0x7fc000000000), ninf->endp 0x3938
 2014-04-15 09:41:57.765: [    CSSD][908056320]clssnmDiscHelper: node 2 clean up, endp (0x3938), init state 5, cur state 5
 2014-04-15 09:41:57.765: [GIPCXCPT][908056320] gipcInternalDissociate: obj 0x7fc024056a70 [0000000000003938] { gipcEndpoint : localAddr 'gipcha://racnode1:nm2_racnode-cluster/da4b-8718-50e0-b51', remoteAddr 'gipcha://racnode2:7b5c-4672-87e8-565', numPend 1, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 1, wobj 0x7fc02402dd90, sendp (nil)flags 0x138606, usrFlags 0x0 } not associated with any container, ret gipcretFail (1)
 2014-04-15 09:41:57.765: [GIPCXCPT][908056320] gipcDissociateF [clssnmDiscHelper : clssnm.c : 3485]: EXCEPTION[ ret gipcretFail (1) ]  failed to dissociate obj 0x7fc024056a70 [0000000000003938] { gipcEndpoint : localAddr 'gipcha://racnode1:nm2_racnode-cluster/da4b-8718-50e0-b51', remoteAddr 'gipcha://racnode2:7b5c-4672-87e8-565', numPend 1, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 1, wobj 0x7fc02402dd90, sendp (nil)flags 0x138606, usrFlags 0x0 }, flags 0x0
 2014-04-15 09:41:57.765: [    CSSD][904902400]clssnmDoSyncUpdate: Initiating sync 291818318
 2014-04-15 09:41:57.765: [    CSSD][904902400]clssscCompareSwapEventValue: changed NMReconfigInProgress  val 1, from -1, changes 7
 2014-04-15 09:41:57.765: [    CSSD][904902400]clssnmDoSyncUpdate: local disk timeout set to 27000 ms, remote disk timeout set to 27000
 2014-04-15 09:41:57.765: [    CSSD][904902400]clssnmDoSyncUpdate: new values for local disk timeout and remote disk timeout will take effect when the sync is completed.
 2014-04-15 09:41:57.765: [    CSSD][903325440]clssnmDiscEndp: gipcDestroy 0x3938
 2014-04-15 09:41:57.765: [    CSSD][904902400]clssnmDoSyncUpdate: Starting cluster reconfig with incarnation 291818318
 2014-04-15 09:41:57.765: [    CSSD][904902400]clssnmSetupAckWait: Ack message type (11)
 2014-04-15 09:41:57.765: [    CSSD][904902400]clssnmSetupAckWait: node(1) is ALIVE
 2014-04-15 09:41:57.765: [    CSSD][904902400]clssnmSendSync: syncSeqNo(291818318), indicating EXADATA fence initialization complete
 2014-04-15 09:41:57.765: [    CSSD][904902400]List of nodes that have ACKed my sync: NULL
 2014-04-15 09:41:57.765: [    CSSD][904902400]clssnmSendSync: syncSeqNo(291818318)
 2014-04-15 09:41:57.765: [    CSSD][903325440]clssnmHandleSync: Node racnode1, number 1, is EXADATA fence capable
 2014-04-15 09:41:57.765: [    CSSD][904902400]clssnmWaitForAcks: Ack message type(11), ackCount(1)
 2014-04-15 09:41:57.765: [    CSSD][903325440]clssscUpdateEventValue: NMReconfigInProgress  val 1, changes 8
 2014-04-15 09:41:57.765: [    CSSD][903325440]clssnmHandleSync: local disk timeout set to 27000 ms, remote disk timeout set to 27000
 2014-04-15 09:41:57.765: [    CSSD][903325440]clssnmHandleSync: initleader 1 newleader 1
 2014-04-15 09:41:57.765: [    CSSD][903325440]clssnmQueueClientEvent:  Sending Event(2), type 2, incarn 291818317
 2014-04-15 09:41:57.765: [    CSSD][903325440]clssnmQueueClientEvent: Node[1] state = 3, birth = 291818316, unique = 1396323475
 2014-04-15 09:41:57.765: [    CSSD][903325440]clssnmQueueClientEvent: Node[2] state = 5, birth = 291818317, unique = 1396323841
 2014-04-15 09:41:57.765: [    CSSD][903325440]clssnmHandleSync: Acknowledging sync: src[1] srcName[racnode1] seq[11] sync[291818318]
 2014-04-15 09:41:57.765: [    CSSD][903325440]clssnmSendAck: node 1, racnode1, syncSeqNo(291818318) type(11)
 2014-04-15 09:41:57.765: [    CSSD][903325440]clssnmHandleAck: Received ack type 11 from node racnode1, number 1, with seq 0 for sync 291818318, waiting for 0 acks
 2014-04-15 09:41:57.765: [    CSSD][904902400]clssnmSendSync: syncSeqNo(291818318), indicating EXADATA fence initialization complete
 2014-04-15 09:41:57.765: [    CSSD][941274880]clssgmStartNMMon: node 1 active, birth 291818316
 2014-04-15 09:41:57.765: [    CSSD][941274880]clssgmStartNMMon: node 2 active, birth 291818317
 2014-04-15 09:41:57.765: [    CSSD][941274880]NMEVENT_SUSPEND [00][00][00][06]
 2014-04-15 09:41:57.765: [    CSSD][941274880]clssgmCompareSwapEventValue: changed CmInfo State  val 5, from 11, changes 21
 2014-04-15 09:41:57.765: [    CSSD][941274880]clssgmSuspendAllGrocks: Issue SUSPEND
 2014-04-15 09:41:57.765: [    CSSD][904902400]List of nodes that have ACKed my sync: 1
 2014-04-15 09:41:57.765: [    CSSD][904902400]clssnmWaitForAcks: done, syncseq(291818318), msg type(11)
 2014-04-15 09:41:57.765: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERAperaXDB) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
 2014-04-15 09:41:57.765: [    CSSD][904902400]clssnmSetMinMaxVersion:node1  product/protocol (11.2/1.4)
 2014-04-15 09:41:57.766: [    CSSD][904902400]clssnmSetMinMaxVersion: properties common to all nodes: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17
 2014-04-15 09:41:57.766: [    CSSD][904902400]clssnmSetMinMaxVersion: min product/protocol (11.2/1.4)
 2014-04-15 09:41:57.766: [    CSSD][904902400]clssnmSetMinMaxVersion: max product/protocol (11.2/1.4)
 2014-04-15 09:41:57.766: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(IG+ASMSYS$USERS) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
 2014-04-15 09:41:57.766: [    CSSD][904902400]clssnmNeedConfReq: No configuration to change
 2014-04-15 09:41:57.766: [    CSSD][904902400]clssnmDoSyncUpdate: Terminating node 2, racnode2, misstime(30000) state(5)
 2014-04-15 09:41:57.766: [    CSSD][904902400]clssnmDoSyncUpdate: Wait for 0 vote ack(s)
 2014-04-15 09:41:57.766: [    CSSD][904902400]clssnmCheckDskInfo: Checking disk info...
 2014-04-15 09:41:57.766: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERASYS$USERS) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
 2014-04-15 09:41:57.766: [    CSSD][904902400]clssnmCheckSplit: Node 2, racnode2, is alive, DHB (1397526116, 1203324554) more than disk timeout of 27000 after the last NHB (1397526085, 1203293884)
 2014-04-15 09:41:57.766: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(crs_version) count(3) master(0) event(2), incarn 3, mbrc 3, to member 0, events 0x20, state 0x0
 2014-04-15 09:41:57.766: [    CSSD][904902400]clssnmCheckDskInfo: My cohort: 1
 2014-04-15 09:41:57.766: [    CSSD][904902400]clssnmRemove: Start
 2014-04-15 09:41:57.766: [    CSSD][904902400](:CSSNM00007:)clssnmrRemoveNode: Evicting node 2, racnode2, from the cluster in incarnation 291818318, node birth incarnation 291818317, death incarnation 291818318, stateflags 0x234000 uniqueness value 1396323841
 2014-04-15 09:41:57.766: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(crs_version) count(3) master(0) event(2), incarn 3, mbrc 3, to member 1, events 0x0, state 0x0
 2014-04-15 09:41:57.767: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(CRF-) count(4) master(0) event(2), incarn 4, mbrc 4, to member 0, events 0x38, state 0x0
 2014-04-15 09:41:57.767: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(CRF-) count(4) master(0) event(2), incarn 4, mbrc 4, to member 1, events 0x38, state 0x0
 2014-04-15 09:41:57.767: [ default][904902400]kgzf_gen_node_reid2: generated reid cid=a2998261b3ccff8abf36841a04ffe27b,icin=291818316,nmn=2,lnid=291818317,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100
 2014-04-15 09:41:57.767: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(CLSN.ONSPROC.MASTER) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0xa0, state 0x0
 2014-04-15 09:41:57.767: [    CSSD][904902400]clssnmrFenceSage: Fenced node racnode2, number 2, with EXADATA, handle 0
 2014-04-15 09:41:57.767: [    CSSD][904902400]clssnmSendShutdown: req to node 2, kill time 1203764884
 2014-04-15 09:41:57.767: [    CSSD][904902400]clssnmsendmsg: not connected to node 2
 2014-04-15 09:41:57.767: [    CSSD][904902400]clssnmSendShutdown: Send to node 2 failed
 2014-04-15 09:41:57.767: [    CSSD][904902400]clssnmWaitOnEvictions: Start
 2014-04-15 09:41:57.767: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(DB+ASM) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x68, state 0x0
 2014-04-15 09:41:57.767: [    CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526116, 1203324554, 1201720), seedhbimpd TRUE
 2014-04-15 09:41:57.767: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(DG+ASM) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
 2014-04-15 09:41:57.767: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(IG+ASMSYS$BACKGROUND) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
 2014-04-15 09:41:57.767: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(DBPERA) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x68, state 0x0
 2014-04-15 09:41:57.767: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(VT+ASM) count(2) master(1) event(2), incarn 6, mbrc 2, to member 1, events 0x60, state 0x0
 2014-04-15 09:41:57.767: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERASYS$BACKGROUND) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
 2014-04-15 09:41:57.767: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(DG_FRA) count(2) master(1) event(2), incarn 5, mbrc 2, to member 1, events 0x4, state 0x0
 2014-04-15 09:41:57.767: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(DG+ASM0) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
 2014-04-15 09:41:57.768: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(GR+GCR1) count(4) master(0) event(2), incarn 26, mbrc 4, to member 0, events 0x280, state 0x0
 2014-04-15 09:41:57.768: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(GR+GCR1) count(4) master(0) event(2), incarn 26, mbrc 4, to member 2, events 0x280, state 0x0
 2014-04-15 09:41:57.768: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(DG_CRS) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x4, state 0x0
 2014-04-15 09:41:57.768: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(DGPERA-) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
 2014-04-15 09:41:57.768: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(DGPERA0) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
 2014-04-15 09:41:57.768: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(DG_CRS1) count(2) master(1) event(2), incarn 5, mbrc 2, to member 1, events 0x4, state 0x0
 2014-04-15 09:41:57.768: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(DG_DATA) count(2) master(1) event(2), incarn 5, mbrc 2, to member 1, events 0x4, state 0x0
 2014-04-15 09:41:57.768: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(CLSN.RLB.pera.MASTER) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0xa0, state 0x0
 2014-04-15 09:41:57.768: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(CLSFRAME) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
 2014-04-15 09:41:57.768: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(EVMDMAIN) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
 2014-04-15 09:41:57.768: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(CRSDMAIN) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
 2014-04-15 09:41:57.768: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(EVMDMAIN2) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
 2014-04-15 09:41:57.769: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERAALL) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
 2014-04-15 09:41:57.769: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(CTSSGROUP) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
 2014-04-15 09:41:57.769: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(CLSN.AQPROC.pera.MASTER) count(2) master(2) event(2), incarn 2, mbrc 2, to member 1, events 0xa0, state 0x0
 2014-04-15 09:41:57.769: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERApera) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
 2014-04-15 09:41:57.769: [    CSSD][941274880]clssgmQueueGrockEvent: groupName(ocr_racnode-cluster) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x78, state 0x0
 2014-04-15 09:41:57.769: [    CSSD][941274880]clssgmSuspendAllGrocks: done
 2014-04-15 09:41:57.769: [    CSSD][941274880]clssgmCompareSwapEventValue: changed CmInfo State  val 2, from 5, changes 22
 2014-04-15 09:41:57.769: [    CSSD][941274880]clssgmUpdateEventValue: ConnectedNodes  val 291818317, changes 7
 2014-04-15 09:41:57.769: [    CSSD][941274880]clssgmCleanupNodeContexts():  cleaning up nodes, rcfg(291818317)
 2014-04-15 09:41:57.769: [    CSSD][941274880]clssgmCleanupNodeContexts():  successful cleanup of nodes rcfg(291818317)
 2014-04-15 09:41:57.769: [GIPCHAUP][920860416] gipchaUpperDisconnect: initiated discconnect umsg 0x7fc024054c60 { msg 0x7fc0240565c8, ret gipcretRequestPending (15), flags 0x2 }, msg 0x7fc0240565c8 { type gipchaMsgTypeDisconnect (5), srcCid 00000000-000038ed, dstCid 00000000-0000059b }, endp 0x7fc0240553a0 [00000000000038ed] { gipchaEndpoint : port 'nm2_racnode-cluster/da4b-8718-50e0-b51b', peer 'racnode2:7b5c-4672-87e8-565a', srcCid 00000000-000038ed,  dstCid 00000000-0000059b, numSend 29, maxSend 100, groupListType 2, hagroup 0x2021010, usrFlags 0x4000, flags 0x21c }
 2014-04-15 09:41:57.769: [    CSSD][941274880]clssgmStartNMMon:  completed node cleanup
 2014-04-15 09:41:57.769: [    CSSD][909633280]clssgmUpdateEventValue: HoldRequest  val 1, changes 5
 2014-04-15 09:41:57.775: [    CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
 2014-04-15 09:41:57.775: [    CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
 2014-04-15 09:41:57.867: [    CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526116, 1203324554, 1201720), seedhbimpd TRUE
 2014-04-15 09:41:57.934: [    CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194016, LATS 1203765044, lastSeqNo 1194015, uniqueness 1396323841, timestamp 1397526117/1203325474
 2014-04-15 09:41:57.934: [    CSSD][914388736]clssnmvDiskEvict: Kill block write, file /dev/asm-diskb flags 0x00010004, kill block unique 1396323841, stamp 1203764884/1203764884
 2014-04-15 09:41:57.934: [    CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
 2014-04-15 09:41:58.035: [    CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
 2014-04-15 09:41:58.135: [    CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
 2014-04-15 09:41:58.149: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest,  hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
 2014-04-15 09:41:58.205: [    CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526118/1203765314
 2014-04-15 09:41:58.235: [    CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
 2014-04-15 09:41:58.335: [    CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
 2014-04-15 09:41:58.435: [    CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
 2014-04-15 09:41:58.535: [    CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
 2014-04-15 09:41:58.551: [    CSSD][916129536]clssnmvDiskKillCheck: not evicted, file /dev/asm-diskb flags 0x00000000, kill block unique 0, my unique 1396323475
 2014-04-15 09:41:58.636: [    CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
 

更多详情见请继续阅读下一页的精彩内容：