RAC上是vmware vspere的虚拟机上有两个OEL6.3的虚拟机，上面跑的库是11.2.0.4
RAC上是vmware vspere的虚拟机上有两个OEL6.3的虚拟机，，上面跑的库是11.2.0.4
发现节点2挂掉了
1.检查节点1的alert
[root@racnode1 racnode1]# pwd
/u01/apps/grid/gridhome/11.2.0/grid/log/racnode1
[root@racnode1 racnode1]# tail -1000 alertracnode1.log
2014-04-15 09:41:20.815:
[crsd(27311)]CRS-2765:Resource 'ora.net1.network' has failed on server 'racnode1'.
2014-04-15 09:41:42.760:
[cssd(26972)]CRS-1612:Network communication with node racnode2 (2) missing for 50% of timeout interval. Removal of this node from cluster in 15.000 seconds
2014-04-15 09:41:50.763:
[cssd(26972)]CRS-1611:Network communication with node racnode2 (2) missing for 75% of timeout interval. Removal of this node from cluster in 7.000 seconds
2014-04-15 09:41:54.764:
[cssd(26972)]CRS-1610:Network communication with node racnode2 (2) missing for 90% of timeout interval. Removal of this node from cluster in 3.000 seconds
2014-04-15 09:41:57.766:
[cssd(26972)]CRS-1607:Node racnode2 is being evicted in cluster incarnation 291818318; details at (:CSSNM00007:) in /u01/apps/grid/gridhome/11.2.0/grid/log/racnode1/cssd/ocssd.log.
2014-04-15 09:42:06.052:
[cssd(26972)]CRS-1625:Node racnode2, number 2, was manually shut down
2014-04-15 09:42:06.059:
[cssd(26972)]CRS-1601:CSSD Reconfiguration complete. Active nodes are racnode1 .
2014-04-15 09:42:06.950:
[crsd(27311)]CRS-5504:Node down event reported for node 'racnode2'.
2014-04-15 09:42:24.882:
[crsd(27311)]CRS-2773:Server 'racnode2' has been removed from pool 'Generic'.
2014-04-15 09:42:24.882:
[crsd(27311)]CRS-2773:Server 'racnode2' has been removed from pool 'ora.pera'.
[root@racnode1 racnode1]#
节点2被驱逐时间：2014-04-15 09:41:57
2.查看ocssd.log
# more /u01/apps/grid/gridhome/11.2.0/grid/log/racnode1/cssd/ocssd.log |grep "2014-04-15 09:41"
2014-04-15 09:41:23.620: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:23.621: [ CSSD][906479360]clssnmSendingThread: sent 4 status msgs to all nodes
2014-04-15 09:41:28.758: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:28.758: [ CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
2014-04-15 09:41:33.145: [GIPCHGEN][919283456] gipchaInterfaceFail: marking interface failing 0x7fc01821d4c0 { host '', haName 'CSS_racnode-cluster', local (nil), ip '172.168.1.11:52955', subnet '172.168.1.0', mask '255.255.255.0', mac '00-50-56-a1-7b-e2', ifname 'eth1', numRef 1, numFail 0, idxBoot 0, flags 0x184d }
2014-04-15 09:41:33.760: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:33.760: [GIPCHGEN][920860416] gipchaInterfaceFail: marking interface failing 0x7fc024040a20 { host 'racnode2', haName 'CSS_racnode-cluster', local 0x7fc01821d4c0, ip '172.168.1.12:60678', subnet '172.168.1.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 4, flags 0x6 }
2014-04-15 09:41:33.760: [ CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
2014-04-15 09:41:33.760: [GIPCHGEN][920860416] gipchaInterfaceDisable: disabling interface 0x7fc01821d4c0 { host '', haName 'CSS_racnode-cluster', local (nil), ip '172.168.1.11:52955', subnet '172.168.1.0', mask '255.255.255.0', mac '00-50-56-a1-7b-e2', ifname 'eth1', numRef 0, numFail 1, idxBoot 0, flags 0x19cd }
2014-04-15 09:41:33.760: [GIPCHGEN][920860416] gipchaInterfaceDisable: disabling interface 0x7fc024040a20 { host 'racnode2', haName 'CSS_racnode-cluster', local 0x7fc01821d4c0, ip '172.168.1.12:60678', subnet '172.168.1.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 4, flags 0x86 }
2014-04-15 09:41:33.761: [GIPCHALO][920860416] gipchaLowerCleanInterfaces: performing cleanup of disabled interface 0x7fc024040a20 { host 'racnode2', haName 'CSS_racnode-cluster', local 0x7fc01821d4c0, ip '172.168.1.12:60678', subnet '172.168.1.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 4, flags 0xa6 }
2014-04-15 09:41:33.885: [GIPCHDEM][920860416] gipchaWorkerCleanInterface: performing cleanup of disabled interface 0x7fc01821d4c0 { host '', haName 'CSS_racnode-cluster', local (nil), ip '172.168.1.11:52955', subnet '172.168.1.0', mask '255.255.255.0', mac '00-50-56-a1-7b-e2', ifname 'eth1', numRef 0, numFail 0, idxBoot 0, flags 0x19ed }
2014-04-15 09:41:34.145: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:37.993: [GIPCHALO][920860416] gipchaLowerProcessNode: no valid interfaces found to node for 5240 ms, node 0x7fc018226540 { host 'racnode2', haName 'CSS_racnode-cluster', srcLuid 8bbba732-d4368a71, dstLuid 0fc9914c-ec29f220 numInf 0, contigSeq 1787047, lastAck 1786995, lastValidAck 1787046, sendSeq [1786996 : 1787014], createTime 1683284, sentRegister 1, localMonitor 1, flags 0x2408 }
2014-04-15 09:41:38.762: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:38.762: [ CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
2014-04-15 09:41:40.146: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:42.760: [ CSSD][908056320]clssnmPollingThread: node racnode2 (2) at 50% heartbeat fatal, removal in 15.000 seconds
2014-04-15 09:41:42.760: [ CSSD][908056320]clssnmPollingThread: node racnode2 (2) is impending reconfig, flag 2294796, misstime 15000
2014-04-15 09:41:42.760: [ CSSD][908056320]clssnmPollingThread: local diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig status(1)
2014-04-15 09:41:42.764: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:42.764: [ CSSD][906479360]clssnmSendingThread: sent 4 status msgs to all nodes
2014-04-15 09:41:43.764: [GIPCHALO][920860416] gipchaLowerProcessNode: no valid interfaces found to node for 11010 ms, node 0x7fc018226540 { host 'racnode2', haName 'CSS_racnode-cluster', srcLuid 8bbba732-d4368a71, dstLuid 0fc9914c-ec29f220 numInf 0, contigSeq 1787047, lastAck 1786995, lastValidAck 1787046, sendSeq [1786996 : 1787025], createTime 1683284, sentRegister 1, localMonitor 1, flags 0x2408 }
2014-04-15 09:41:43.822: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526103/1203750934
2014-04-15 09:41:43.906: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1193997, LATS 1203751024, lastSeqNo 1175771, uniqueness 1396323841, timestamp 1397526102/1203310914
2014-04-15 09:41:45.102: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526105/1203752214
2014-04-15 09:41:45.772: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526105/1203752884
2014-04-15 09:41:46.147: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:46.181: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1193999, LATS 1203753294, lastSeqNo 1193997, uniqueness 1396323841, timestamp 1397526104/1203313164
2014-04-15 09:41:46.782: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526106/1203753894
2014-04-15 09:41:47.120: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194001, LATS 1203754234, lastSeqNo 1193999, uniqueness 1396323841, timestamp 1397526106/1203314834
2014-04-15 09:41:47.622: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526107/1203754734
2014-04-15 09:41:47.765: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:47.765: [ CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
2014-04-15 09:41:48.004: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194002, LATS 1203755124, lastSeqNo 1194001, uniqueness 1396323841, timestamp 1397526107/1203315684
2014-04-15 09:41:48.507: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526108/1203755624
2014-04-15 09:41:48.774: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194003, LATS 1203755894, lastSeqNo 1194002, uniqueness 1396323841, timestamp 1397526108/1203316514
2014-04-15 09:41:49.213: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526109/1203756324
2014-04-15 09:41:49.766: [GIPCHALO][920860416] gipchaLowerProcessNode: no valid interfaces found to node for 17010 ms, node 0x7fc018226540 { host 'racnode2', haName 'CSS_racnode-cluster', srcLuid 8bbba732-d4368a71, dstLuid 0fc9914c-ec29f220 numInf 0, contigSeq 1787047, lastAck 1786995, lastValidAck 1787046, sendSeq [1786996 : 1787037], createTime 1683284, sentRegister 1, localMonitor 1, flags 0x2408 }
2014-04-15 09:41:49.893: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194004, LATS 1203757004, lastSeqNo 1194003, uniqueness 1396323841, timestamp 1397526109/1203317334
2014-04-15 09:41:49.903: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526109/1203757014
2014-04-15 09:41:50.528: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194005, LATS 1203757644, lastSeqNo 1194004, uniqueness 1396323841, timestamp 1397526109/1203317964
2014-04-15 09:41:50.548: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526110/1203757664
2014-04-15 09:41:50.763: [ CSSD][908056320]clssnmPollingThread: node racnode2 (2) at 75% heartbeat fatal, removal in 7.000 seconds
2014-04-15 09:41:50.875: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194006, LATS 1203757994, lastSeqNo 1194005, uniqueness 1396323841, timestamp 1397526110/1203318664
2014-04-15 09:41:51.113: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526111/1203758224
2014-04-15 09:41:51.396: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194007, LATS 1203758514, lastSeqNo 1194006, uniqueness 1396323841, timestamp 1397526110/1203319204
2014-04-15 09:41:51.692: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526111/1203758804
2014-04-15 09:41:52.148: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:52.294: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194008, LATS 1203759404, lastSeqNo 1194007, uniqueness 1396323841, timestamp 1397526111/1203319754
2014-04-15 09:41:52.305: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526112/1203759424
2014-04-15 09:41:52.767: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:52.767: [ CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
2014-04-15 09:41:52.772: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194009, LATS 1203759884, lastSeqNo 1194008, uniqueness 1396323841, timestamp 1397526112/1203320364
2014-04-15 09:41:52.974: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526112/1203760084
2014-04-15 09:41:53.793: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194010, LATS 1203760904, lastSeqNo 1194009, uniqueness 1396323841, timestamp 1397526112/1203321034
2014-04-15 09:41:53.934: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526113/1203761044
2014-04-15 09:41:54.401: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194012, LATS 1203761514, lastSeqNo 1194010, uniqueness 1396323841, timestamp 1397526114/1203322404
2014-04-15 09:41:54.494: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526114/1203761604
2014-04-15 09:41:54.764: [ CSSD][908056320]clssnmPollingThread: node racnode2 (2) at 90% heartbeat fatal, removal in 3.000 seconds, seedhbimpd 1
2014-04-15 09:41:55.044: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526115/1203762154
2014-04-15 09:41:55.134: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194013, LATS 1203762254, lastSeqNo 1194012, uniqueness 1396323841, timestamp 1397526114/1203322964
2014-04-15 09:41:55.640: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526115/1203762754
2014-04-15 09:41:55.768: [GIPCHALO][920860416] gipchaLowerProcessNode: no valid interfaces found to node for 23010 ms, node 0x7fc018226540 { host 'racnode2', haName 'CSS_racnode-cluster', srcLuid 8bbba732-d4368a71, dstLuid 0fc9914c-ec29f220 numInf 0, contigSeq 1787047, lastAck 1786995, lastValidAck 1787046, sendSeq [1786996 : 1787048], createTime 1683284, sentRegister 1, localMonitor 1, flags 0x2408 }
2014-04-15 09:41:56.498: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526116/1203763614
2014-04-15 09:41:57.145: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194015, LATS 1203764254, lastSeqNo 1194013, uniqueness 1396323841, timestamp 1397526116/1203324554
2014-04-15 09:41:57.645: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526117/1203764764
2014-04-15 09:41:57.764: [ CSSD][908056320]clssnmPollingThread: Removal started for node racnode2 (2), flags 0x23040c, state 3, wt4c 0
2014-04-15 09:41:57.764: [ CSSD][908056320]clssnmMarkNodeForRemoval: node 2, racnode2 marked for removal
2014-04-15 09:41:57.765: [ CSSD][908056320]clssnmDiscHelper: racnode2, node(2) connection failed, endp (0x3938), probe(0x7fc000000000), ninf->endp 0x3938
2014-04-15 09:41:57.765: [ CSSD][908056320]clssnmDiscHelper: node 2 clean up, endp (0x3938), init state 5, cur state 5
2014-04-15 09:41:57.765: [GIPCXCPT][908056320] gipcInternalDissociate: obj 0x7fc024056a70 [0000000000003938] { gipcEndpoint : localAddr 'gipcha://racnode1:nm2_racnode-cluster/da4b-8718-50e0-b51', remoteAddr 'gipcha://racnode2:7b5c-4672-87e8-565', numPend 1, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 1, wobj 0x7fc02402dd90, sendp (nil)flags 0x138606, usrFlags 0x0 } not associated with any container, ret gipcretFail (1)
2014-04-15 09:41:57.765: [GIPCXCPT][908056320] gipcDissociateF [clssnmDiscHelper : clssnm.c : 3485]: EXCEPTION[ ret gipcretFail (1) ] failed to dissociate obj 0x7fc024056a70 [0000000000003938] { gipcEndpoint : localAddr 'gipcha://racnode1:nm2_racnode-cluster/da4b-8718-50e0-b51', remoteAddr 'gipcha://racnode2:7b5c-4672-87e8-565', numPend 1, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 1, wobj 0x7fc02402dd90, sendp (nil)flags 0x138606, usrFlags 0x0 }, flags 0x0
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmDoSyncUpdate: Initiating sync 291818318
2014-04-15 09:41:57.765: [ CSSD][904902400]clssscCompareSwapEventValue: changed NMReconfigInProgress val 1, from -1, changes 7
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmDoSyncUpdate: local disk timeout set to 27000 ms, remote disk timeout set to 27000
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmDoSyncUpdate: new values for local disk timeout and remote disk timeout will take effect when the sync is completed.
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmDiscEndp: gipcDestroy 0x3938
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmDoSyncUpdate: Starting cluster reconfig with incarnation 291818318
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmSetupAckWait: Ack message type (11)
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmSetupAckWait: node(1) is ALIVE
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmSendSync: syncSeqNo(291818318), indicating EXADATA fence initialization complete
2014-04-15 09:41:57.765: [ CSSD][904902400]List of nodes that have ACKed my sync: NULL
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmSendSync: syncSeqNo(291818318)
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmHandleSync: Node racnode1, number 1, is EXADATA fence capable
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmWaitForAcks: Ack message type(11), ackCount(1)
2014-04-15 09:41:57.765: [ CSSD][903325440]clssscUpdateEventValue: NMReconfigInProgress val 1, changes 8
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmHandleSync: local disk timeout set to 27000 ms, remote disk timeout set to 27000
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmHandleSync: initleader 1 newleader 1
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmQueueClientEvent: Sending Event(2), type 2, incarn 291818317
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmQueueClientEvent: Node[1] state = 3, birth = 291818316, unique = 1396323475
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmQueueClientEvent: Node[2] state = 5, birth = 291818317, unique = 1396323841
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmHandleSync: Acknowledging sync: src[1] srcName[racnode1] seq[11] sync[291818318]
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmSendAck: node 1, racnode1, syncSeqNo(291818318) type(11)
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmHandleAck: Received ack type 11 from node racnode1, number 1, with seq 0 for sync 291818318, waiting for 0 acks
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmSendSync: syncSeqNo(291818318), indicating EXADATA fence initialization complete
2014-04-15 09:41:57.765: [ CSSD][941274880]clssgmStartNMMon: node 1 active, birth 291818316
2014-04-15 09:41:57.765: [ CSSD][941274880]clssgmStartNMMon: node 2 active, birth 291818317
2014-04-15 09:41:57.765: [ CSSD][941274880]NMEVENT_SUSPEND [00][00][00][06]
2014-04-15 09:41:57.765: [ CSSD][941274880]clssgmCompareSwapEventValue: changed CmInfo State val 5, from 11, changes 21
2014-04-15 09:41:57.765: [ CSSD][941274880]clssgmSuspendAllGrocks: Issue SUSPEND
2014-04-15 09:41:57.765: [ CSSD][904902400]List of nodes that have ACKed my sync: 1
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmWaitForAcks: done, syncseq(291818318), msg type(11)
2014-04-15 09:41:57.765: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERAperaXDB) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmSetMinMaxVersion:node1 product/protocol (11.2/1.4)
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmSetMinMaxVersion: properties common to all nodes: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmSetMinMaxVersion: min product/protocol (11.2/1.4)
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmSetMinMaxVersion: max product/protocol (11.2/1.4)
2014-04-15 09:41:57.766: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(IG+ASMSYS$USERS) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmNeedConfReq: No configuration to change
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmDoSyncUpdate: Terminating node 2, racnode2, misstime(30000) state(5)
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmDoSyncUpdate: Wait for 0 vote ack(s)
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmCheckDskInfo: Checking disk info...
2014-04-15 09:41:57.766: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERASYS$USERS) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmCheckSplit: Node 2, racnode2, is alive, DHB (1397526116, 1203324554) more than disk timeout of 27000 after the last NHB (1397526085, 1203293884)
2014-04-15 09:41:57.766: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(crs_version) count(3) master(0) event(2), incarn 3, mbrc 3, to member 0, events 0x20, state 0x0
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmCheckDskInfo: My cohort: 1
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmRemove: Start
2014-04-15 09:41:57.766: [ CSSD][904902400](:CSSNM00007:)clssnmrRemoveNode: Evicting node 2, racnode2, from the cluster in incarnation 291818318, node birth incarnation 291818317, death incarnation 291818318, stateflags 0x234000 uniqueness value 1396323841
2014-04-15 09:41:57.766: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(crs_version) count(3) master(0) event(2), incarn 3, mbrc 3, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CRF-) count(4) master(0) event(2), incarn 4, mbrc 4, to member 0, events 0x38, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CRF-) count(4) master(0) event(2), incarn 4, mbrc 4, to member 1, events 0x38, state 0x0
2014-04-15 09:41:57.767: [ default][904902400]kgzf_gen_node_reid2: generated reid cid=a2998261b3ccff8abf36841a04ffe27b,icin=291818316,nmn=2,lnid=291818317,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CLSN.ONSPROC.MASTER) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0xa0, state 0x0
2014-04-15 09:41:57.767: [ CSSD][904902400]clssnmrFenceSage: Fenced node racnode2, number 2, with EXADATA, handle 0
2014-04-15 09:41:57.767: [ CSSD][904902400]clssnmSendShutdown: req to node 2, kill time 1203764884
2014-04-15 09:41:57.767: [ CSSD][904902400]clssnmsendmsg: not connected to node 2
2014-04-15 09:41:57.767: [ CSSD][904902400]clssnmSendShutdown: Send to node 2 failed
2014-04-15 09:41:57.767: [ CSSD][904902400]clssnmWaitOnEvictions: Start
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DB+ASM) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x68, state 0x0
2014-04-15 09:41:57.767: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526116, 1203324554, 1201720), seedhbimpd TRUE
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DG+ASM) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(IG+ASMSYS$BACKGROUND) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DBPERA) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x68, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(VT+ASM) count(2) master(1) event(2), incarn 6, mbrc 2, to member 1, events 0x60, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERASYS$BACKGROUND) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DG_FRA) count(2) master(1) event(2), incarn 5, mbrc 2, to member 1, events 0x4, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DG+ASM0) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(GR+GCR1) count(4) master(0) event(2), incarn 26, mbrc 4, to member 0, events 0x280, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(GR+GCR1) count(4) master(0) event(2), incarn 26, mbrc 4, to member 2, events 0x280, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DG_CRS) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x4, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DGPERA-) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DGPERA0) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DG_CRS1) count(2) master(1) event(2), incarn 5, mbrc 2, to member 1, events 0x4, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DG_DATA) count(2) master(1) event(2), incarn 5, mbrc 2, to member 1, events 0x4, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CLSN.RLB.pera.MASTER) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0xa0, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CLSFRAME) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(EVMDMAIN) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CRSDMAIN) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(EVMDMAIN2) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERAALL) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CTSSGROUP) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CLSN.AQPROC.pera.MASTER) count(2) master(2) event(2), incarn 2, mbrc 2, to member 1, events 0xa0, state 0x0
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERApera) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(ocr_racnode-cluster) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x78, state 0x0
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmSuspendAllGrocks: done
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmCompareSwapEventValue: changed CmInfo State val 2, from 5, changes 22
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmUpdateEventValue: ConnectedNodes val 291818317, changes 7
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmCleanupNodeContexts(): cleaning up nodes, rcfg(291818317)
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmCleanupNodeContexts(): successful cleanup of nodes rcfg(291818317)
2014-04-15 09:41:57.769: [GIPCHAUP][920860416] gipchaUpperDisconnect: initiated discconnect umsg 0x7fc024054c60 { msg 0x7fc0240565c8, ret gipcretRequestPending (15), flags 0x2 }, msg 0x7fc0240565c8 { type gipchaMsgTypeDisconnect (5), srcCid 00000000-000038ed, dstCid 00000000-0000059b }, endp 0x7fc0240553a0 [00000000000038ed] { gipchaEndpoint : port 'nm2_racnode-cluster/da4b-8718-50e0-b51b', peer 'racnode2:7b5c-4672-87e8-565a', srcCid 00000000-000038ed, dstCid 00000000-0000059b, numSend 29, maxSend 100, groupListType 2, hagroup 0x2021010, usrFlags 0x4000, flags 0x21c }
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmStartNMMon: completed node cleanup
2014-04-15 09:41:57.769: [ CSSD][909633280]clssgmUpdateEventValue: HoldRequest val 1, changes 5
2014-04-15 09:41:57.775: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:57.775: [ CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
2014-04-15 09:41:57.867: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526116, 1203324554, 1201720), seedhbimpd TRUE
2014-04-15 09:41:57.934: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194016, LATS 1203765044, lastSeqNo 1194015, uniqueness 1396323841, timestamp 1397526117/1203325474
2014-04-15 09:41:57.934: [ CSSD][914388736]clssnmvDiskEvict: Kill block write, file /dev/asm-diskb flags 0x00010004, kill block unique 1396323841, stamp 1203764884/1203764884
2014-04-15 09:41:57.934: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
2014-04-15 09:41:58.035: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
2014-04-15 09:41:58.135: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
2014-04-15 09:41:58.149: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:58.205: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526118/1203765314
2014-04-15 09:41:58.235: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
2014-04-15 09:41:58.335: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
2014-04-15 09:41:58.435: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
2014-04-15 09:41:58.535: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
2014-04-15 09:41:58.551: [ CSSD][916129536]clssnmvDiskKillCheck: not evicted, file /dev/asm-diskb flags 0x00000000, kill block unique 0, my unique 1396323475
2014-04-15 09:41:58.636: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
更多详情见请继续阅读下一页的精彩内容：
What are stored procedures in MySQL?May 01, 2025 am 12:27 AMStored procedures are precompiled SQL statements in MySQL for improving performance and simplifying complex operations. 1. Improve performance: After the first compilation, subsequent calls do not need to be recompiled. 2. Improve security: Restrict data table access through permission control. 3. Simplify complex operations: combine multiple SQL statements to simplify application layer logic.
How does query caching work in MySQL?May 01, 2025 am 12:26 AMThe working principle of MySQL query cache is to store the results of SELECT query, and when the same query is executed again, the cached results are directly returned. 1) Query cache improves database reading performance and finds cached results through hash values. 2) Simple configuration, set query_cache_type and query_cache_size in MySQL configuration file. 3) Use the SQL_NO_CACHE keyword to disable the cache of specific queries. 4) In high-frequency update environments, query cache may cause performance bottlenecks and needs to be optimized for use through monitoring and adjustment of parameters.
What are the advantages of using MySQL over other relational databases?May 01, 2025 am 12:18 AMThe reasons why MySQL is widely used in various projects include: 1. High performance and scalability, supporting multiple storage engines; 2. Easy to use and maintain, simple configuration and rich tools; 3. Rich ecosystem, attracting a large number of community and third-party tool support; 4. Cross-platform support, suitable for multiple operating systems.
How do you handle database upgrades in MySQL?Apr 30, 2025 am 12:28 AMThe steps for upgrading MySQL database include: 1. Backup the database, 2. Stop the current MySQL service, 3. Install the new version of MySQL, 4. Start the new version of MySQL service, 5. Recover the database. Compatibility issues are required during the upgrade process, and advanced tools such as PerconaToolkit can be used for testing and optimization.
What are the different backup strategies you can use for MySQL?Apr 30, 2025 am 12:28 AMMySQL backup policies include logical backup, physical backup, incremental backup, replication-based backup, and cloud backup. 1. Logical backup uses mysqldump to export database structure and data, which is suitable for small databases and version migrations. 2. Physical backups are fast and comprehensive by copying data files, but require database consistency. 3. Incremental backup uses binary logging to record changes, which is suitable for large databases. 4. Replication-based backup reduces the impact on the production system by backing up from the server. 5. Cloud backups such as AmazonRDS provide automation solutions, but costs and control need to be considered. When selecting a policy, database size, downtime tolerance, recovery time, and recovery point goals should be considered.
What is MySQL clustering?Apr 30, 2025 am 12:28 AMMySQLclusteringenhancesdatabaserobustnessandscalabilitybydistributingdataacrossmultiplenodes.ItusestheNDBenginefordatareplicationandfaulttolerance,ensuringhighavailability.Setupinvolvesconfiguringmanagement,data,andSQLnodes,withcarefulmonitoringandpe
How do you optimize database schema design for performance in MySQL?Apr 30, 2025 am 12:27 AMOptimizing database schema design in MySQL can improve performance through the following steps: 1. Index optimization: Create indexes on common query columns, balancing the overhead of query and inserting updates. 2. Table structure optimization: Reduce data redundancy through normalization or anti-normalization and improve access efficiency. 3. Data type selection: Use appropriate data types, such as INT instead of VARCHAR, to reduce storage space. 4. Partitioning and sub-table: For large data volumes, use partitioning and sub-table to disperse data to improve query and maintenance efficiency.
How can you optimize MySQL performance?Apr 30, 2025 am 12:26 AMTooptimizeMySQLperformance,followthesesteps:1)Implementproperindexingtospeedupqueries,2)UseEXPLAINtoanalyzeandoptimizequeryperformance,3)Adjustserverconfigurationsettingslikeinnodb_buffer_pool_sizeandmax_connections,4)Usepartitioningforlargetablestoi
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Linux new version
SublimeText3 Linux latest version
VSCode Windows 64-bit Download
A free and powerful IDE editor launched by Microsoft
SAP NetWeaver Server Adapter for Eclipse
Integrate Eclipse with SAP NetWeaver application server.
mPDF
mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),
