ORA-15064 错误解决

整理一下前段时间遇到的ORA-15064错误。有一套测试环境是11gR2的rac使用ASM，应用在测试时发现一个节点宕机了，检查了一下alert日

整理一下前段时间遇到的ORA-15064错误。

有一套测试环境是11gR2的rac使用ASM，应用在测试时发现一个节点宕机了，检查了一下alert日志如下错误：

Mon Jun 04 23:27:32 2012

NOTE: ASMB terminating
Errors in file /soft/Oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
Errors in file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
ASMB (ospid: 13828564): terminating the instance due to error 15064
Termination issued to instance processes. Waiting for the processes to exit
Mon Jun 04 23:27:43 2012
Instance termination failed to kill one or more processes

Instance terminated by ASMB, pid = 13828564

通过错误可知，改错无与ASM有关，，应该是ASM意外关闭才导致的Instance关闭。

错误的trace文件内容如下：

[pt1rac2@root]# cat /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc

Trace file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /soft/oracle/product/11.2.0.2/dbh
System name:    AIX
Node name:      pt1rac2
Release:        1
Version:        6
Machine:        00F770E14C00
Instance name: pt1db2
Redo thread mounted by this instance: 0
Oracle process number: 27
Unix process pid: 13828564, image: oracle@pt1rac2 (ASMB)




*** 2012-06-04 16:19:14.483
*** SESSION ID:(2998.1) 2012-06-04 16:19:14.483
*** CLIENT ID:() 2012-06-04 16:19:14.483
*** SERVICE NAME:() 2012-06-04 16:19:14.483
*** MODULE NAME:() 2012-06-04 16:19:14.483
*** ACTION NAME:() 2012-06-04 16:19:14.483
 
NOTE: initiating MARK startup 


*** 2012-06-04 16:19:18.290
Mode of disk 0 (DATA_0000) changing from 0x7 to 0x7


*** 2012-06-04 23:07:12.286
Mode of disk 0 (DATA_0000) changing from 0x7 to 0x7


*** 2012-06-04 23:27:32.492
NOTE: ASMB terminating
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID: 
Session ID: 23 Serial number: 1161
error 15064 detected in background process
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID: 
Session ID: 23 Serial number: 1161
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+644
----- End of Abridged Call Stack Trace -----


*** 2012-06-04 23:27:32.854
ASMB (ospid: 13828564): terminating the instance due to error 15064


*** 2012-06-04 23:27:43.387
Instance termination failed to kill one or more processes

可知的确是由于ASM意外关闭才导致的instance关闭，ASM意外关闭的原因继续查。

准备生产一下awr报告看一下，结果发现由于数据关闭了，没有awr可看。

一下数据说明数据库被重启过几次，且经过与应用人员确认，这些重启的时间点他们都在做压力测试，因此初步怀疑是和性能有关，继续查。

                                                        Snap
Instance     DB Name        Snap Id    Snap Started    Level
------------ ------------ --------- ------------------ -----
pt1db2       PT1DB              503 02 Jun 2012 16:00      1
                                504 02 Jun 2012 17:00      1


                                505 04 Jun 2012 09:43      1


                                506 04 Jun 2012 11:00      1
                                507 04 Jun 2012 12:00      1
                                508 04 Jun 2012 13:00      1
                                509 04 Jun 2012 14:00      1
                                510 04 Jun 2012 15:00      1
                                511 04 Jun 2012 16:00      1
                                512 04 Jun 2012 17:00      1
                                513 04 Jun 2012 18:00      1
                                514 04 Jun 2012 19:00      1
                                515 04 Jun 2012 20:00      1
                                516 04 Jun 2012 21:00      1
                                517 04 Jun 2012 22:00      1
                                518 04 Jun 2012 23:00      1


                                522 05 Jun 2012 01:20      1
                                523 05 Jun 2012 01:30      1
                                524 05 Jun 2012 01:45      1

先查看ocssd.log日志

2012-06-04 23:26:02.274: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 113598 msecs
2012-06-04 23:26:03.126: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:03.126: [    CSSD][1029]clssgmAllocProc: (11328bcd0) allocated
2012-06-04 23:26:03.127: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11328bcd0 - 0,1,2,3,4
2012-06-04 23:26:03.127: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2ea97f) proc(11328bcd0) pid(15335558) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:03.127: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:03.274: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:03.274: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:03.309: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:03.309: [    CSSD][1029]clssgmAllocProc: (1132d30d0) allocated
2012-06-04 23:26:03.310: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132d30d0 - 0,1,2,3,4
2012-06-04 23:26:03.310: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2ea9fa) proc(1132d30d0) pid(9306434/9306434) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:03.310: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:03.315: [    CSSD][1029]clssscSelect: cookie accept request 1132d30d0
2012-06-04 23:26:03.315: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132d30d0
2012-06-04 23:26:03.316: [    CSSD][1029]clssgmRegisterClient: proc(57/1132d30d0), client(1/11313ef70)
2012-06-04 23:26:03.316: [    CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1

。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。

2012-06-04 23:26:29.658: [    CSSD][1029]clssgmDeadProc: proc 112f32d30
2012-06-04 23:26:29.658: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(112f32d30) con(2ea049) skgpid  ospid 35520704 with 0 clients, refcount 0
2012-06-04 23:26:29.658: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea049
2012-06-04 23:26:29.675: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:29.675: [    CSSD][1029]clssgmAllocProc: (113242130) allocated
2012-06-04 23:26:29.676: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 113242130 - 0,1,2,3,4
2012-06-04 23:26:29.676: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eadef) proc(113242130) pid(10879472) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:29.676: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.226: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.226: [    CSSD][1029]clssgmAllocProc: (1132a3390) allocated
2012-06-04 23:26:30.227: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132a3390 - 0,1,2,3,4
2012-06-04 23:26:30.227: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eae64) proc(1132a3390) pid(35520706) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.228: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.542: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.542: [    CSSD][1029]clssgmAllocProc: (1132ad030) allocated
2012-06-04 23:26:30.544: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132ad030 - 0,1,2,3,4
2012-06-04 23:26:30.544: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eaede) proc(1132ad030) pid(8585490/8585490) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.544: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.553: [    CSSD][1029]clssscSelect: cookie accept request 1132ad030
2012-06-04 23:26:30.553: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132ad030
2012-06-04 23:26:30.553: [    CSSD][1029]clssgmRegisterClient: proc(63/1132ad030), client(1/1132ad370)
2012-06-04 23:26:30.554: [    CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:26:30.554: [    CSSD][1029]clssgmQueueShare: (1132ad510) target global grock DG_DATA member 1 type 1 queued from client (1132ad370), global grock DG_DATA, refcount 15
2012-06-04 23:26:30.554: [    CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 15
2012-06-04 23:26:30.557: [    CSSD][1029]clssscSelect: cookie accept request 1132ad030
2012-06-04 23:26:30.557: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132ad030
2012-06-04 23:26:30.557: [    CSSD][1029]clssgmRegisterClient: proc(63/1132ad030), client(2/1132a3d90)
2012-06-04 23:26:30.558: [    CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:26:30.558: [    CSSD][1029]clssgmQueueShare: (1132ad910) target global grock DB+ASM member 1 type 1 queued from client (1132a3d90), global grock DB+ASM, refcount 17
2012-06-04 23:26:30.558: [    CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 17
2012-06-04 23:26:30.636: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.637: [    CSSD][1029]clssgmAllocProc: (11333ba70) allocated
2012-06-04 23:26:30.638: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11333ba70 - 0,1,2,3,4
2012-06-04 23:26:30.638: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eaf6f) proc(11333ba70) pid(8913188/8913188) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.638: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.650: [    CSSD][1029]clssscSelect: cookie accept request 11333ba70
2012-06-04 23:26:30.650: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11333ba70
2012-06-04 23:26:30.650: [    CSSD][1029]clssgmRegisterClient: proc(64/11333ba70), client(1/112e8e550)
2012-06-04 23:26:30.652: [    CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:26:30.652: [    CSSD][1029]clssgmQueueShare: (11328ed90) target global grock DG_DATA member 1 type 1 queued from client (112e8e550), global grock DG_DATA, refcount 16
2012-06-04 23:26:30.652: [    CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 16
2012-06-04 23:26:30.766: [    CSSD][1029]clssscSelect: cookie accept request 11333ba70
2012-06-04 23:26:30.766: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11333ba70
2012-06-04 23:26:30.766: [    CSSD][1029]clssgmRegisterClient: proc(64/11333ba70), client(2/11328eff0)
2012-06-04 23:26:30.772: [    CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:26:30.772: [    CSSD][1029]clssgmQueueShare: (11328f190) target global grock DB+ASM member 1 type 1 queued from client (11328eff0), global grock DB+ASM, refcount 18
2012-06-04 23:26:30.773: [    CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 18
2012-06-04 23:26:31.382: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:31.382: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:34.394: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 145718 msecs
2012-06-04 23:26:35.392: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:35.392: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:36.396: [    CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 47678 msecs
2012-06-04 23:26:38.571: [    CSSD][2834]clssnmvDiskCheck: (/dev/rhdisk1) No I/O completed after 75% maximum time, 200000 ms, will be considered unusable in 49104 ms
2012-06-04 23:26:39.408: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:39.409: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:42.413: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 153738 msecs
2012-06-04 23:26:43.413: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:43.413: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:44.414: [    CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 55696 msecs
2012-06-04 23:26:47.421: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:47.421: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:50.433: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 161758 msecs
2012-06-04 23:26:51.431: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:51.431: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:52.439: [    CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 63721 msecs
2012-06-04 23:26:55.442: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:55.442: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:58.445: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 169771 msecs
2012-06-04 23:26:59.455: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:59.455: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:00.493: [    CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 71776 msecs
2012-06-04 23:27:03.470: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:03.470: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:06.473: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 177799 msecs
2012-06-04 23:27:07.475: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:07.476: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:08.192: [    CSSD][1029]clssgmDeadProc: proc 112f35f50
2012-06-04 23:27:08.192: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(112f35f50) con(2ea340) skgpid  ospid 10551654 with 0 clients, refcount 0
2012-06-04 23:27:08.192: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea340
2012-06-04 23:27:08.192: [    CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 79475 msecs
2012-06-04 23:27:08.498: [    CSSD][1029]clssgmDeadProc: proc 1131b4b10
2012-06-04 23:27:08.498: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(1131b4b10) con(2ea3c4) skgpid  ospid 10355074 with 0 clients, refcount 0
2012-06-04 23:27:08.498: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea3c4
2012-06-04 23:27:08.569: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.569: [    CSSD][1029]clssgmAllocProc: (1131fe670) allocated
2012-06-04 23:27:08.571: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1131fe670 - 0,1,2,3,4
2012-06-04 23:27:08.571: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb1f8) proc(1131fe670) pid(10551656) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.571: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.637: [    CSSD][2834]clssnmvDiskCheck: (/dev/rhdisk1) No I/O completed after 90% maximum time, 200000 ms, will be considered unusable in 19038 ms
2012-06-04 23:27:08.809: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.810: [    CSSD][1029]clssgmAllocProc: (1132593f0) allocated
2012-06-04 23:27:08.811: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132593f0 - 0,1,2,3,4
2012-06-04 23:27:08.811: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb278) proc(1132593f0) pid(10355076) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.811: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.855: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.855: [    CSSD][1029]clssgmAllocProc: (113259730) allocated
2012-06-04 23:27:08.857: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 113259730 - 0,1,2,3,4
2012-06-04 23:27:08.858: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb2ec) proc(113259730) pid(8192508/8192508) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.858: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.867: [    CSSD][1029]clssscSelect: cookie accept request 113259730
2012-06-04 23:27:08.867: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 113259730
2012-06-04 23:27:08.867: [    CSSD][1029]clssgmRegisterClient: proc(65/113259730), client(1/112f35f50)
2012-06-04 23:27:08.868: [    CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:08.868: [    CSSD][1029]clssgmQueueShare: (11334b430) target global grock DG_DATA member 1 type 1 queued from client (112f35f50), global grock DG_DATA, refcount 17
2012-06-04 23:27:08.868: [    CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 17
2012-06-04 23:27:08.872: [    CSSD][1029]clssscSelect: cookie accept request 113259730
2012-06-04 23:27:08.872: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 113259730
2012-06-04 23:27:08.872: [    CSSD][1029]clssgmRegisterClient: proc(65/113259730), client(2/113215ed0)
2012-06-04 23:27:08.874: [    CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:08.874: [    CSSD][1029]clssgmQueueShare: (11334b690) target global grock DB+ASM member 1 type 1 queued from client (113215ed0), global grock DB+ASM, refcount 19
2012-06-04 23:27:08.874: [    CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 19
2012-06-04 23:27:09.000: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:09.000: [    CSSD][1029]clssgmAllocProc: (11334b8f0) allocated
2012-06-04 23:27:09.001: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11334b8f0 - 0,1,2,3,4
2012-06-04 23:27:09.001: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb37d) proc(11334b8f0) pid(8061184/8061184) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:09.001: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:09.010: [    CSSD][1029]clssscSelect: cookie accept request 11334b8f0
2012-06-04 23:27:09.010: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11334b8f0
2012-06-04 23:27:09.010: [    CSSD][1029]clssgmRegisterClient: proc(66/11334b8f0), client(1/1131b5310)
2012-06-04 23:27:09.011: [    CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:09.011: [    CSSD][1029]clssgmQueueShare: (11334bc30) target global grock DG_DATA member 1 type 1 queued from client (1131b5310), global grock DG_DATA, refcount 18
2012-06-04 23:27:09.011: [    CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 18
2012-06-04 23:27:09.014: [    CSSD][1029]clssscSelect: cookie accept request 11334b8f0
2012-06-04 23:27:09.014: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11334b8f0
2012-06-04 23:27:09.014: [    CSSD][1029]clssgmRegisterClient: proc(66/11334b8f0), client(2/11334be90)
2012-06-04 23:27:09.015: [    CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:09.015: [    CSSD][1029]clssgmQueueShare: (113350230) target global grock DB+ASM member 1 type 1 queued from client (11334be90), global grock DB+ASM, refcount 20
2012-06-04 23:27:09.015: [    CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 20
2012-06-04 23:27:11.480: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:11.480: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:14.494: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 185820 msecs
2012-06-04 23:27:15.500: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:15.500: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:16.504: [    CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 87787 msecs
2012-06-04 23:27:19.519: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:19.519: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:20.285: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:20.285: [    CSSD][1029]clssgmAllocProc: (1132b9290) allocated
2012-06-04 23:27:20.294: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b9290 - 0,1,2,3,4
2012-06-04 23:27:20.294: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb49d) proc(1132b9290) pid(8257826) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:20.294: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:20.299: [    CSSD][1029]clssnmGetNodeNumber: pt1rac2 
2012-06-04 23:27:20.457: [    CSSD][1029]clssgmDeadProc: proc 1132b9290
2012-06-04 23:27:20.457: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(1132b9290) con(2eb49d) skgpid  ospid 8257826 with 0 clients, refcount 0
2012-06-04 23:27:20.457: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2eb49d
2012-06-04 23:27:20.999: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:20.999: [    CSSD][1029]clssgmAllocProc: (1132b2650) allocated
2012-06-04 23:27:21.000: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b2650 - 0,1,2,3,4
2012-06-04 23:27:21.000: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb51a) proc(1132b2650) pid(7733742) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:21.000: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:21.005: [    CSSD][1029]clssnmGetNodeNumber: pt1rac1 
2012-06-04 23:27:21.289: [    CSSD][1029]clssgmDeadProc: proc 1132b2650
2012-06-04 23:27:21.290: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(1132b2650) con(2eb51a) skgpid  ospid 7733742 with 0 clients, refcount 0
2012-06-04 23:27:21.290: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2eb51a
2012-06-04 23:27:22.001: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 193327 msecs
2012-06-04 23:27:23.533: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:23.534: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:24.945: [    CSSD][1029]clssgmDeadProc: proc 113248030
2012-06-04 23:27:24.945: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(113248030) con(2ea626) skgpid  ospid 9699686 with 0 clients, refcount 0
2012-06-04 23:27:24.945: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea626
2012-06-04 23:27:25.142: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:25.143: [    CSSD][1029]clssgmAllocProc: (1132b2650) allocated
2012-06-04 23:27:25.143: [    CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 96425 msecs
2012-06-04 23:27:25.144: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b2650 - 1,2,3,4
2012-06-04 23:27:25.144: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb5c3) proc(1132b2650) pid(9699688) version 11:2:1:4, properties: 1,2,3,4
2012-06-04 23:27:25.144: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:25.323: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:25.324: [    CSSD][1029]clssgmAllocProc: (1132b7e50) allocated
2012-06-04 23:27:25.326: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b7e50 - 0,1,2,3,4
2012-06-04 23:27:25.326: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb63a) proc(1132b7e50) pid(7733746/7733746) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:25.326: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:25.485: [    CSSD][1029]clssscSelect: cookie accept request 1132b7e50
2012-06-04 23:27:25.485: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132b7e50
2012-06-04 23:27:25.485: [    CSSD][1029]clssgmRegisterClient: proc(67/1132b7e50), client(1/113153d90)
2012-06-04 23:27:25.487: [    CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:25.487: [    CSSD][1029]clssgmQueueShare: (113338450) target global grock DG_DATA member 1 type 1 queued from client (113153d90), global grock DG_DATA, refcount 19
2012-06-04 23:27:25.487: [    CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 19
2012-06-04 23:27:25.495: [    CSSD][1029]clssscSelect: cookie accept request 1132b7e50
2012-06-04 23:27:25.495: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132b7e50
2012-06-04 23:27:25.495: [    CSSD][1029]clssgmRegisterClient: proc(67/1132b7e50), client(2/1132b42b0)
2012-06-04 23:27:25.497: [    CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:25.497: [    CSSD][1029]clssgmQueueShare: (11334c0f0) target global grock DB+ASM member 1 type 1 queued from client (1132b42b0), global grock DB+ASM, refcount 21
2012-06-04 23:27:25.497: [    CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 21
2012-06-04 23:27:27.542: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:27.542: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:27.680: [    CSSD][2834](:CSSNM00058:)clssnmvDiskCheck: No I/O completions for 200006 ms for voting file /dev/rhdisk1)
2012-06-04 23:27:27.693: [    CSSD][2834]clssnmvDiskAvailabilityChange: voting file /dev/rhdisk1 now offline
2012-06-04 23:27:27.693: [    CSSD][2834](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
2012-06-04 23:27:27.694: [    CSSD][2834]###################################
2012-06-04 23:27:27.694: [    CSSD][2834]clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
2012-06-04 23:27:27.694: [    CSSD][2834]###################################
2012-06-04 23:27:27.694: [    CSSD][2834](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
2012-06-04 23:27:27.695: [    CSSD][2834]


----- Call Stack Trace -----
2012-06-04 23:27:27.723: [    CSSD][2834]calling              call     entry                argument values in hex      
2012-06-04 23:27:27.723: [    CSSD][2834]location             type     point                (? means dubious value)     
2012-06-04 23:27:27.723: [    CSSD][2834]-------------------- -------- -------------------- ----------------------------
2012-06-04 23:27:27.724: [    CSSD][2834]clssscExit()+692     bl       100067114            67800000678 ? 1000EB40C ?
2012-06-04 23:27:27.724: [    CSSD][2834]                                                   110091190 ? 900000000BA4D14 ?
2012-06-04 23:27:27.724: [    CSSD][2834]                                                   000000000 ? 013960031 ?
2012-06-04 23:27:27.724: [    CSSD][2834]                                                   013960031 ? 000000000 ?
2012-06-04 23:27:27.724: [    CSSD][2834]clssnmvDiskCheck()+  call     clssscExit()         111E38B90 ? 200000002 ?
2012-06-04 23:27:27.724: [    CSSD][2834]2576                                               000000001 ? 1000F52A0 ?
2012-06-04 23:27:27.724: [    CSSD][2834]                                                   110091190 ? 013960031 ?
2012-06-04 23:27:27.724: [    CSSD][2834]                                                   013960031 ? 000000000 ?
2012-06-04 23:27:27.724: [    CSSD][2834]clssnmvDiskPingMoni  call     clssnmvDiskCheck()   111E38B90 ? 110BF7FB0 ?
2012-06-04 23:27:27.724: [    CSSD][2834]torThread()+392                                    112468680 ? 000000000 ?
2012-06-04 23:27:27.724: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   013960031 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]clssscthrdmain()+20  call     clssnmvDiskPingMoni  111E38B90 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]8                             torThread()          000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]_pthread_body()+240  call     clssscthrdmain()     111E38B90 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]fffffffffffffffc     call     _pthread_body()      000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.726: [    CSSD][2834] 
2012-06-04 23:27:27.726: [    CSSD][2834]--------------------- Binary Stack Dump ---------------------
2012-06-04 23:27:27.726: [    CSSD][2834] 
2012-06-04 23:27:27.726: [    CSSD][2834]========== FRAME [1] (clssscExit()+692 -> 100067114) ==========
2012-06-04 23:27:27.726: [    CSSD][2834]defined by frame pointers 0x1124683f0  and 0x112468260
2012-06-04 23:27:27.726: [    CSSD][2834]CALL TYPE: bl   ERROR SIGNALED: no   CALLER: clssscExit
2012-06-04 23:27:27.726: [    CSSD][2834]
Dump of memory from 0x112468260 to 0x1124683f0
2012-06-04 23:27:27.726: [    CSSD][2834]112468260 00000001 124683F0 44000224 000003D7  [.....F..D..$....]
2012-06-04 23:27:27.726: [    CSSD][2834]112468270 00000001 000031D8 00000000 00000083  [......1.........]
2012-06-04 23:27:27.726: [    CSSD][2834]112468280 00000000 00000029 00000001 100182B0  [.......)........]
2012-06-04 23:27:27.726: [    CSSD][2834]112468290 00000678 00000678 00000001 000EB40C  [...x...x........]
2012-06-04 23:27:27.726: [    CSSD][2834]1124682A0 00000001 10091190 09000000 00BA4D14  [..............M.]
2012-06-04 23:27:27.726: [    CSSD][2834]1124682B0 00000000 00000000 00000000 13960031  [...............1]
2012-06-04 23:27:27.727: [    CSSD][2834]1124682C0 00000000 13960031 00000000 00000000  [.......1........]
2012-06-04 23:27:27.727: [    CSSD][2834]1124682D0 00000001 1008FF50 00000001 1002BD70  [.......P.......p]
2012-06-04 23:27:27.727: [    CSSD][2834]1124682E0 00000000 00000001 43525300 12468368  [........CRS..F.h]
2012-06-04 23:27:27.727: [    CSSD][2834]1124682F0 00000001 00000000 00000001 100181F0  [................]
2012-06-04 23:27:27.727: [    CSSD][2834]112468300 00000001 10018208 00000001 100181F0  [................]
2012-06-04 23:27:27.727: [    CSSD][2834]112468310 00000001 10018208 00000001 11E38B90  [................]
2012-06-04 23:27:27.727: [    CSSD][2834]112468320 00000001 11E38B90 00000000 00000000  [................]
2012-06-04 23:27:27.727: [    CSSD][2834]112468330 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.728: [    CSSD][2834]        Repeat 3 times
2012-06-04 23:27:27.728: [    CSSD][2834]112468370 00000000 00000000 00000000 13960031  [...............1]
2012-06-04 23:27:27.728: [    CSSD][2834]112468380 09000000 00C27160 00000000 00000000  [......q`........]
2012-06-04 23:27:27.728: [    CSSD][2834]112468390 00000001 1002BD70 00000001 117B6590  [.......p.....{e.]
2012-06-04 23:27:27.728: [    CSSD][2834]1124683A0 00000001 00000001 00000000 00000000  [................]
2012-06-04 23:27:27.728: [    CSSD][2834]1124683B0 00000000 00000000 00000646 00000646  [...........F...F]
2012-06-04 23:27:27.728: [    CSSD][2834]1124683C0 00000001 12468428 00000001 117C6C84  [.....F.(.....|l.]
2012-06-04 23:27:27.728: [    CSSD][2834]1124683D0 00000001 117C6C84 00000001 1008FF50  [.....|l........P]
2012-06-04 23:27:27.728: [    CSSD][2834]1124683E0 00000000 00000000 00000001 000F24E8  [..............$.]
2012-06-04 23:27:27.728: [    CSSD][2834]========== FRAME [2] (clssnmvDiskCheck()+2576 -> clssscExit()) ==========
2012-06-04 23:27:27.728: [    CSSD][2834]defined by frame pointers 0x112468600  and 0x1124683f0
2012-06-04 23:27:27.729: [    CSSD][2834]CALL TYPE: call   ERROR SIGNALED: no   CALLER: clssnmvDiskCheck
2012-06-04 23:27:27.729: [    CSSD][2834]
Dump of memory from 0x1124683f0 to 0x112468600
2012-06-04 23:27:27.729: [    CSSD][2834]1124683F0 00000001 12468600 09001000 A0688640  [.....F.......h.@]
2012-06-04 23:27:27.729: [    CSSD][2834]112468400 00000001 0002CD94 4400B22B 00000000  [........D..+....]
2012-06-04 23:27:27.729: [    CSSD][2834]112468410 00000001 001058B8 00000001 100182B0  [......X.........]
2012-06-04 23:27:27.729: [    CSSD][2834]112468420 00000001 11E38B90 00000002 00000002  [................]
2012-06-04 23:27:27.729: [    CSSD][2834]112468430 00000000 00000001 00000001 000F52A0  [..............R.]
2012-06-04 23:27:27.729: [    CSSD][2834]112468440 00000001 10091190 00000000 13960031  [...............1]
2012-06-04 23:27:27.729: [    CSSD][2834]112468450 00000000 13960031 00000000 00000000  [.......1........]
2012-06-04 23:27:27.730: [    CSSD][2834]112468460 00000001 1008FF50 00000001 118508F0  [.......P........]
2012-06-04 23:27:27.730: [    CSSD][2834]112468470 00000001 117B6590 8C91440C 00030D46  [.....{e...D....F]
2012-06-04 23:27:27.730: [    CSSD][2834]112468480 00000001 12C508B0 00000000 00030D40  [...............@]
2012-06-04 23:27:27.730: [    CSSD][2834]112468490 00010001 9DD81012 D0E54F97 BFEE0F2C  [..........O....,]
2012-06-04 23:27:27.730: [    CSSD][2834]1124684A0 AF8AA88C 00000000 2200088B 00000000  [........".......]
2012-06-04 23:27:27.730: [    CSSD][2834]1124684B0 09000000 014F0814 09001000 A0688640  [.....O.......h.@]
2012-06-04 23:27:27.730: [    CSSD][2834]1124684C0 00030D40 1045F530 00000004 00000004  [...@.E.0........]
2012-06-04 23:27:27.730: [    CSSD][2834]1124684D0 00000001 00105DE8 00000001 000F2BD4  [......].......+.]
2012-06-04 23:27:27.730: [    CSSD][2834]1124684E0 00000001 000F2BD4 00030D40 000F24E8  [......+....@..$.]
2012-06-04 23:27:27.731: [    CSSD][2834]1124684F0 00000001 00000001 2200022B 00000000  [........"..+....]
2012-06-04 23:27:27.731: [    CSSD][2834]112468500 00000000 00000000 FFFFFFF5 000F24E8  [..............$.]
2012-06-04 23:27:27.731: [    CSSD][2834]112468510 00000001 12468600 09001000 A0688640  [.....F.......h.@]
2012-06-04 23:27:27.731: [    CSSD][2834]112468520 00000001 00070DFC 00000001 117C6C84  [.............|l.]
2012-06-04 23:27:27.731: [    CSSD][2834]112468530 00000001 117C6C84 00000001 100182B0  [.....|l.........]
2012-06-04 23:27:27.731: [    CSSD][2834]112468540 00000001 1045F530 00000004 00000004  [.....E.0........]
2012-06-04 23:27:27.731: [    CSSD][2834]112468550 00000001 00105DE8 00000000 00000009  [......].........]
2012-06-04 23:27:27.731: [    CSSD][2834]112468560 00000000 000003E8 00000000 000003E8  [................]
2012-06-04 23:27:27.731: [    CSSD][2834]112468570 00000000 13960031 00000000 00000040  [.......1.......@]
2012-06-04 23:27:27.732: [    CSSD][2834]112468580 00000001 1008FF50 00000001 117B6590  [.......P.....{e.]
2012-06-04 23:27:27.732: [    CSSD][2834]112468590 00000001 1002BD70 00000001 117B65A0  [.......p.....{e.]
2012-06-04 23:27:27.732: [    CSSD][2834]1124685A0 00000001 117B66A0 FFFFFFB2 000003D4  [.....{f.........]
2012-06-04 23:27:27.732: [    CSSD][2834]1124685B0 00000001 117C6C84 00000001 117C6C84  [.....|l......|l.]
2012-06-04 23:27:27.732: [    CSSD][2834]        Repeat 1 times
2012-06-04 23:27:27.732: [    CSSD][2834]1124685D0 00000001 000F2BD4 00000001 000F2BD4  [......+.......+.]
2012-06-04 23:27:27.732: [    CSSD][2834]1124685E0 00000000 00000000 00000001 12C508B0  [................]
2012-06-04 23:27:27.732: [    CSSD][2834]1124685F0 00000000 00000000 00000001 000F24E8  [..............$.]
2012-06-04 23:27:27.732: [    CSSD][2834]========== FRAME [3] (clssnmvDiskPingMonitorThread()+392 -> clssnmvDiskCheck()) ==========
2012-06-04 23:27:27.732: [    CSSD][2834]defined by frame pointers 0x1124686f0  and 0x112468600
2012-06-04 23:27:27.732: [    CSSD][2834]CALL TYPE: call   ERROR SIGNALED: no   CALLER: clssnmvDiskPingMonitorThread
2012-06-04 23:27:27.732: [    CSSD][2834]
Dump of memory from 0x112468600 to 0x1124686f0
2012-06-04 23:27:27.733: [    CSSD][2834]112468600 00000001 124686F0 00000000 00000000  [.....F..........]
2012-06-04 23:27:27.733: [    CSSD][2834]112468610 00000001 0002CFAC 09001000 A0688640  [.............h.@]
2012-06-04 23:27:27.733: [    CSSD][2834]112468620 00000000 00000000 00000001 100182B0  [................]
2012-06-04 23:27:27.733: [    CSSD][2834]112468630 00000001 11E38B90 00000001 10BF7FB0  [................]
2012-06-04 23:27:27.733: [    CSSD][2834]112468640 00000001 12468680 00000000 00000000  [.....F..........]
2012-06-04 23:27:27.733: [    CSSD][2834]112468650 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.733: [    CSSD][2834]112468660 00000000 13960031 00000000 00000000  [.......1........]
2012-06-04 23:27:27.733: [    CSSD][2834]112468670 00000001 10BF7FB0 00000001 117B6590  [.............{e.]
2012-06-04 23:27:27.733: [    CSSD][2834]112468680 00030D40 8C914024 000003E8 100182B0  [...@..@$........]
2012-06-04 23:27:27.733: [    CSSD][2834]112468690 09000000 014F0814 00000000 00000000  [.....O..........]
2012-06-04 23:27:27.734: [    CSSD][2834]1124686A0 00000000 00000000 00000000 00000001  [................]
2012-06-04 23:27:27.734: [    CSSD][2834]1124686B0 000003E8 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.734: [    CSSD][2834]1124686C0 00000000 00000000 000003E8 000003E8  [................]
2012-06-04 23:27:27.734: [    CSSD][2834]1124686D0 000003E8 1008FF50 00000001 1002BD70  [.......P.......p]
2012-06-04 23:27:27.734: [    CSSD][2834]1124686E0 00000001 10BF97D0 00000001 12468A78  [.............F.x]
2012-06-04 23:27:27.734: [    CSSD][2834]========== FRAME [4] (clssscthrdmain()+208 -> clssnmvDiskPingMonitorThread()) ==========
2012-06-04 23:27:27.734: [    CSSD][2834]defined by frame pointers 0x112468790  and 0x1124686f0
2012-06-04 23:27:27.734: [    CSSD][2834]CALL TYPE: call   ERROR SIGNALED: no   CALLER: clssscthrdmain
2012-06-04 23:27:27.734: [    CSSD][2834]
Dump of memory from 0x1124686f0 to 0x112468790
2012-06-04 23:27:27.734: [    CSSD][2834]1124686F0 00000001 12468790 00000000 00000000  [.....F..........]
2012-06-04 23:27:27.734: [    CSSD][2834]112468700 00000001 0000A294 00000000 00000000  [................]
2012-06-04 23:27:27.735: [    CSSD][2834]112468710 00000000 00000000 00000001 100182B0  [................]
2012-06-04 23:27:27.735: [    CSSD][2834]112468720 00000001 11E38B90 00000000 00000000  [................]
2012-06-04 23:27:27.735: [    CSSD][2834]112468730 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.735: [    CSSD][2834]        Repeat 2 times
2012-06-04 23:27:27.735: [    CSSD][2834]112468760 00000001 1008FF50 00000001 1002BD70  [.......P.......p]
2012-06-04 23:27:27.735: [    CSSD][2834]112468770 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.735: [    CSSD][2834]112468780 00000000 00000000 00000001 1008FF50  [...............P]
2012-06-04 23:27:27.735: [    CSSD][2834]========== FRAME [5] (_pthread_body()+240 -> clssscthrdmain()) ==========
2012-06-04 23:27:27.735: [    CSSD][2834]defined by frame pointers 0x112468810  and 0x112468790
2012-06-04 23:27:27.735: [    CSSD][2834]CALL TYPE: call   ERROR SIGNALED: no   CALLER: _pthread_body
2012-06-04 23:27:27.735: [    CSSD][2834]
Dump of memory from 0x112468790 to 0x112468890
2012-06-04 23:27:27.736: [    CSSD][2834]112468790 00000001 12468810 00000000 00000000  [.....F..........]
2012-06-04 23:27:27.736: [    CSSD][2834]1124687A0 09000000 00B7BD54 00000000 00000000  [.......T........]
2012-06-04 23:27:27.736: [    CSSD][2834]1124687B0 00000000 00000000 09001000 A0288350  [.............(.P]
2012-06-04 23:27:27.736: [    CSSD][2834]1124687C0 00000001 11E38B90 00000000 00000000  [................]
2012-06-04 23:27:27.736: [    CSSD][2834]1124687D0 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.737: [    CSSD][2834]  Repeat 11 times
2012-06-04 23:27:27.737: [    CSSD][2834]========== FRAME [6] (fffffffffffffffc -> _pthread_body()) ==========
2012-06-04 23:27:27.737: [    CSSD][2834]defined by frame pointers 0x0  and 0x112468810
2012-06-04 23:27:27.737: [    CSSD][2834]CALL TYPE: call   ERROR SIGNALED: no   CALLER: fffffffffffffffc
2012-06-04 23:27:27.737: [    CSSD][2834]
----- Argument/Register Address Dump -----


2012-06-04 23:27:27.737: [    CSSD][2834]Argument/Register addr=0x1000eb40c.
Dump of memory from 0x1000eb3cc to 0x1000eb50c
2012-06-04 23:27:27.737: [    CSSD][2834]1000EB3C0                            6C206572              [l er]
2012-06-04 23:27:27.737: [    CSSD][2834]1000EB3D0 726F7220 6F636375 72726564 20616E64  [ror occurred and]
2012-06-04 23:27:27.737: [    CSSD][2834]1000EB3E0 20746865 20435353 20646165 6D6F6E20  [ the CSS daemon ]
2012-06-04 23:27:27.737: [    CSSD][2834]1000EB3F0 69732074 65726D69 6E617469 6E672061  [is terminating a]
2012-06-04 23:27:27.737: [    CSSD][2834]1000EB400 626E6F72 6D616C6C 79004942 283A4353  [bnormally.IB(:CS]
2012-06-04 23:27:27.737: [    CSSD][2834]1000EB410 53534330 30303132 3A290049 636C7373  [SSC00012:).Iclss]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB420 73634578 69743A20 53746172 74696E67  [scExit: Starting]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB430 20435253 4420636C 65616E75 70004942  [ CRSD cleanup.IB]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB440 636C7373 73634578 69743A20 43525344  [clssscExit: CRSD]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB450 20636C65 616E7570 20737461 74757320  [ cleanup status ]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB460 25640049 636C7373 73634578 69743A20  [%d.IclssscExit: ]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB470 43525344 20636C65 616E7570 20666169  [CRSD cleanup fai]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB480 6C656420 77697468 20256400 636C7373  [led with %d.clss]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB490 73634578 69743A20 43525344 20636C65  [scExit: CRSD cle]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB4A0 616E7570 20737563 63657373 66756C6C  [anup successfull]
2012-06-04 23:27:27.739: [    CSSD][2834]1000EB4B0 7920636F 6D706C65 74656400 636C7373  [y completed.clss]
2012-06-04 23:27:27.739: [    CSSD][2834]1000EB4C0 73634578 69743A20 61626F72 7420616C  [scExit: abort al]
2012-06-04 23:27:27.739: [    CSSD][2834]1000EB4D0 72656164 79207365 74202564 0049424D  [ready set %d.IBM]
2012-06-04 23:27:27.739: [    CSSD][2834]1000EB4E0 41535345 52542025 73202564 0049424D  [ASSERT %s %d.IBM]
2012-06-04 23:27:27.739: [    CSSD][2834]1000EB4F0 636C7373 73632E63 0049424D 283A4353  [clsssc.c.IBM(:CS]
2012-06-04 23:27:27.739: [    CSSD][2834]1000EB500 53534330 30303133 3A29636C           [SSC00013:)cl]    
2012-06-04 23:27:27.739: [    CSSD][2834]Argument/Register addr=0x110091190.
Dump of memory from 0x110091150 to 0x110091290
2012-06-04 23:27:27.739: [    CSSD][2834]110091150 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.739: [    CSSD][2834]        Repeat 2 times
2012-06-04 23:27:27.739: [    CSSD][2834]110091180 00000000 00000000 00000001 00000001  [................]
2012-06-04 23:27:27.740: [    CSSD][2834]110091190 2F736F66 742F6772 69642F70 726F6475  [/soft/grid/produ]
2012-06-04 23:27:27.740: [    CSSD][2834]1100911A0 63742F31 312E322E 302E322F 6769682F  [ct/11.2.0.2/gih/]
2012-06-04 23:27:27.740: [    CSSD][2834]1100911B0 6C6F672F 70743172 6163322F 63737364  [log/pt1rac2/cssd]
2012-06-04 23:27:27.740: [    CSSD][2834]1100911C0 2F6F6373 73642E6C 6F670000 00000000  [/ocssd.log......]
2012-06-04 23:27:27.740: [    CSSD][2834]1100911D0 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.741: [    CSSD][2834]  Repeat 11 times
2012-06-04 23:27:27.741: [    CSSD][2834]Argument/Register addr=0x900000000ba4d14.
Dump of memory from 0x900000000ba4cd4 to 0x900000000ba4e14
2012-06-04 23:27:27.741: [    CSSD][2834]900000000BA4CD0          38600000 4BFD5608 00000000      [8`..K.V.....]
2012-06-04 23:27:27.741: [    CSSD][2834]900000000BA4CE0 000C0000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.741: [    CSSD][2834]900000000BA4CF0 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.741: [    CSSD][2834]900000000BA4D00 348D8278 41810008 38800000 E8630000  [4..xA...8....c..]
2012-06-04 23:27:27.741: [    CSSD][2834]900000000BA4D10 38A00008 81840230 7C0004AC 398CFFFF  [8......0|...9...]
2012-06-04 23:27:27.741: [    CSSD][2834]900000000BA4D20 E9040110 38E00000 65081000 7D251828  [....8...e...}%.(]
2012-06-04 23:27:27.741: [    CSSD][2834]900000000BA4D30 7C894000 40860014 7CE5192D 40A2FFF0  [|.@.@...|..-@...]
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4D40 91840230 4E800020 7C632A14 4BFD4694  [...0N.. |c*.K.F.]
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4D50 00000000 000C0000 00000000 00000000  [................]
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4D60 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.742: [    CSSD][2834]        Repeat 1 times
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4D80 7D251828 64871000 2C890000 40860014  [}%.(d...,...@...]
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4D90 7CE5192D 4C00012C 4D820020 4BFFFFE4  [|..-L..,M.. K...]
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4DA0 7C862378 7C851A14 7CC53378 4BFD5534  [|.#x|...|.3xK.U4]
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4DB0 00000000 000C0000 00000000 00000000  [................]
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4DC0 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.743: [    CSSD][2834]        Repeat 3 times
2012-06-04 23:27:27.743: [    CSSD][2834]900000000BA4E00 7D251828 64871000 2C890000 40860014  [}%.(d...,...@...]
2012-06-04 23:27:27.743: [    CSSD][2834]900000000BA4E10 7CE5192D                             [|..-]            
2012-06-04 23:27:27.743: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.743: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.744: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.744: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.744: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.745: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.745: [    CSSD][2834]Argument/Register addr=0x111e38b90.
Dump of memory from 0x111e38b50 to 0x111e38c90
2012-06-04 23:27:27.745: [    CSSD][2834]111E38B50 00000001 00000001 8C8FC179 000003E8  [...........y....]
2012-06-04 23:27:27.745: [    CSSD][2834]111E38B60 8C914038 8C913A22 0000000B 00000000  [..@8..:"........]
2012-06-04 23:27:27.745: [    CSSD][2834]111E38B70 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.745: [    CSSD][2834]111E38B80 5B5B5B5B 00000000 00000000 000001E0  [[[[[............]
2012-06-04 23:27:27.745: [    CSSD][2834]111E38B90 00000001 1008FF50 00000001 10BF7FB0  [.......P........]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38BA0 00000001 11E3A320 00000001 11E389A0  [....... ........]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38BB0 00000B12 00000B12 636C7373 6E6D7644  [........clssnmvD]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38BC0 69736B50 696E674D 6F6E6974 6F725468  [iskPingMonitorTh]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38BD0 72656164 00000000 00000000 00000000  [read............]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38BE0 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38BF0 00000000 00000000 00020000 00000000  [................]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38C00 00000001 1045F530 00000001 11E38B90  [.....E.0........]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38C10 00000001 100179B0 00000000 00000000  [......y.........]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38C20 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.747: [    CSSD][2834]        Repeat 4 times
2012-06-04 23:27:27.747: [    CSSD][2834]111E38C70 00000000 00000001 00000000 00000005  [................]
2012-06-04 23:27:27.747: [    CSSD][2834]111E38C80 00000000 00001677 00000000 00000000  [.......w........]
2012-06-04 23:27:27.747: [    CSSD][2834]Argument/Register addr=0x1.
Dump of memory from 0x0 to 0x101
2012-06-04 23:27:27.747: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.748: [    CSSD][2834]        Repeat 15 times
2012-06-04 23:27:27.748: [    CSSD][2834]000000100 7C7243A6                             [|rC.]            
2012-06-04 23:27:27.748: [    CSSD][2834]Argument/Register addr=0x1000f52a0.
Dump of memory from 0x1000f5260 to 0x1000f53a0
2012-06-04 23:27:27.748: [    CSSD][2834]1000F5260 6B436865 636B3A20 41626F72 74696E67  [kCheck: Aborting]
2012-06-04 23:27:27.748: [    CSSD][2834]1000F5270 2C202564 206F6620 25642025 7320766F  [, %d of %d %s vo]
2012-06-04 23:27:27.748: [    CSSD][2834]1000F5280 74696E67 20646973 6B732061 7661696C  [ting disks avail]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F5290 61626C65 2C206E65 65642025 75004942  [able, need %u.IB]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F52A0 283A4353 534E4D30 30303138 3A290049  [(:CSSNM00018:).I]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F52B0 636C7373 6E6D7644 69736B50 696E674D  [clssnmvDiskPingM]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F52C0 6F6E6974 6F725468 72656164 3A20206E  [onitorThread:  n]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F52D0 6F20766F 74696E67 20646973 6B2C0049  [o voting disk,.I]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F52E0 636C7373 6E6D7644 69736B50 4D543A20  [clssnmvDiskPMT: ]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F52F0 736C6565 70696E67 20666F72 20256420  [sleeping for %d ]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F5300 6D730049 636C7373 6E6D7653 63616E43  [ms.IclssnmvScanC]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F5310 6F6D706C 6574696F 6E733A20 4661696C  [ompletions: Fail]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F5320 75726520 2564206E 6F746966 79696E67  [ure %d notifying]
2012-06-04 23:27:27.750: [    CSSD][2834]1000F5330 20636C69 656E7473 206F6620 776F726B  [ clients of work]
2012-06-04 23:27:27.750: [    CSSD][2834]1000F5340 20636F6D 706C6574 696F6E2E 0049424D  [ completion..IBM]
2012-06-04 23:27:27.750: [    CSSD][2834]1000F5350 636C7373 6E6D7653 63616E43 6F6D706C  [clssnmvScanCompl]
2012-06-04 23:27:27.750: [    CSSD][2834]1000F5360 6574696F 6E733A20 636F6D70 6C657465  [etions: complete]
2012-06-04 23:27:27.750: [    CSSD][2834]1000F5370 64202564 20697465 6D730049 636C7373  [d %d items.Iclss]
2012-06-04 23:27:27.750: [    CSSD][2834]1000F5380 6E6D7652 6561644C 65617365 3A20656E  [nmvReadLease: en]
2012-06-04 23:27:27.750: [    CSSD][2834]1000F5390 74657200 636C7373 6E6D7652 6561644C &nbs