Home > Article > Web Front-end > [RAC] Node 1 cannot start due to lack of cssfatal
Environment: AIX5.3+10.2.0.5 RAC
Scenario description: When RAC is restarted after shutting down, the first node cannot start, but the second node starts normally
Troubleshooting process:
1. Try to start the crs service of node 1
root# ./init.crs start crs
2. During the monitoring startup process, crs log
OCSSD.log log:
[CSSD]2014-01-16 09:27:54.730 >USER: Copyright 2014, Oracle version 10.2.0.5.0
[CSSD]2014-01-16 09:27:54.730 >USER: Starting CSS daemon on node nxjcdb1, number1, in cluster crs_dljc
[clsdmt]Listening to(ADDRESS= (PROTOCOL=ipc)(KEY=nxjcdb1DBG_CSSD))
[CSSD]2014-01-16 09:27:54.790 [1]>TRACE: clssscmain: RT queuesetting: ON
[CSSD]2014-01-16 09 :27:55.081 [1]>TRACE: clssscmain: local-only setto false
[CSSD]2014-01-16 09:27:55.349 [1]>TRACE: clssnmReadNodeInfo: addednode 1 (nxjcdb1) to cluster
[ CSSD]2014-01-16 09:27:55.672 [1]>TRACE: clssnmReadNodeInfo: addednode 2 (nxjcdb2) to cluster
[ CSSD]2014-01-16 09:27:55.673 [1]> TRACE: clssnmInitNMInfo:Initialized with unique 1389835674
[CSSD]2014-01-16 09:27:55.704 [1]>TRACE: clssNMInitialize:Initializing with OCR id (1516675067)
[CSSD]2014-01-1 6 09 :27:55.705 [1029] >TRACE: clssnm_skgxninit: HACMP clusterware detected
[ CSSD]2014-01-16 09:27:56.822 [1]>TRACE: clssnmNMInitialize:misscount set to (30)
[ CSSD ]2014-01-16 09:27:56.900 [1]>TRACE: clssnmStartNM: reboottimeset to (3) sec
[ CSSD]2014-01-16 09:27:56.900 [1]>TRACE: clssnmNMInitialize: Networkheartbeat thresholds are: impending reconfig 15000 ms, reconfig start(misscount) 30000 ms
[CSSD]2014-01-16 09:27:57.108 [1]>TRACE: clssnmDiskStateChange: statefrom 1 to 2 disk (0//dev /rlvjc_voting)
[CSSD]2014-01-16 09:27:57.108 [1030]>TRACE: clssnmvDPT: spawned for disk0 (/dev/rlvjc_voting)
[CSSD]2014-01-16 09:27: 1030 //dev/rlvjc_voting)
[CSSD]2014-01-16 09:27:59.164 [1]>ERROR: Internal Error Information:
Category: 1234
Operation: scls_scr_setval
Location: open
Other : cant open file
Dep: 2
[CSSD]2014-01-16 09:27:59.164 [1]>ERROR: clssscSclsFatal: failure 8reading fatal mode
[CSSD]2014-01-16 09:27 :59.164 [1]>ERROR: ################################
[CSSD]2014 -01-16 09:27:59.164 [1]>ERROR: clssscExit: CSSD aborting from thread Main
[ CSSD]2014-01-16 09:27:59.164 [1]>ERROR: ###### ############################
→According to the error message, it is initially determined that OCSSD cannot start because node 1 cannot vote disk.
[ CSSD]--- DUMP GROCK STATE DB ---
[ CSSD]--- END OF GROCK STATE DUMP ---
[ CSSD]2014-01-16 09:27:59.169 [1030]> ;TRACE: clssnmvReadDskHeartbeat:read ALL for Joining
[CSSD]2014-01-16 09:27:59.169 [1030]>TRACE: clssnmvReadDskHeartbeat:node(2) is down. rcfg(2) wrtcnt(126947) LATS( 1038806686) Disk lastSeqNo(126947)
[CSSD]------- Begin Dump -------
[CSSD]
[CSSD]
[CSSD]
[CSSD]
[CSSD]
[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100863c0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00..... ........
[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100863d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ... .............
[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100863e0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ............
[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100863f0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ............
[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086400 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ............
[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086410 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ............
[CSSD]2014-01-16 09:28:00.166 [1 ]>TRACE: 0x110086420 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 .....
[CSSD]2014-01-16 09: 28:00.166 [1]>TRACE: 0x110086430 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 .....
[CSSD]2014- 01-16 09:28:00.166 [1]>TRACE: 0x110086440 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ............
[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086450 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ............. ...
[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086460 00 00 00 01 1008 61 98 - 00 00 00 01 10 c6 0b c0 ......a. ........
[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086470 00 00 00 00 00 0000 01 - 00 00 00 00 00 02 00 03 ... .............
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086480 00 00 00 01 1096 4a b0 - 00 00 00 00 00 00 00 00 ......J.........
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086490 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864a0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864b0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 01 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864c0 00 00 00 00 0000 00 05 - 00 00 00 00 00 00 00 fa ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864e0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864f0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086500 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086510 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086520 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086530 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086540 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086550 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086560 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086570 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086580 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086590 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865a0 00 00 00 00 0000 00 00 - 00 00 00 0e 00 00 00 24 ...............$
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865b0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865c0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865e0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865f0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086600 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086610 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086620 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086630 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086640 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086650 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086660 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086670 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086680 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086690 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866a0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866b0 00 00 10 00 0000 00 97 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866c0 00 00 00 01 105d f6 10 - 00 00 00 01 10 95 ca 50 .....].........P
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866d0 00 00 00 01 1096 0a 70 - 00 00 00 3c 00 00 00 00 .......p...<....
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866e0 00 00 00 01 1096 2a 90 - 00 00 00 01 00 00 00 01 ......*.........
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866f0 00 00 00 28 0000 00 00 - 00 00 00 01 10 00 16 08 ...(............
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086700 00 00 00 00 0000 00 00 - 00 00 00 01 10 4c 1a 90 .............L..
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086710 00 00 00 01 104c 1a 50 - 00 00 00 01 10 08 67 18 .....L.P......g.
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086720 00 00 00 01 1008 67 18 - 00 00 00 00 00 00 00 00 ......g.........
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086730 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086740 00 00 00 01 10cf 76 b0 - 00 00 00 00 00 00 00 00 ......v.........
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086750 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086760 00 00 00 01 10cf 76 b0 - 00 00 00 00 00 00 00 00 ......v.........
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086770 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086780 00 00 01 00 0000 00 00 - 00 00 00 01 10 bd f7 90 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086790 00 00 00 01 104c 17 50 - 00 00 00 00 00 00 00 00 .....L.P........
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867a0 00 00 00 00 0000 00 00 - 00 00 00 03 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867b0 00 00 00 01 1000 09 78 - 00 00 00 01 00 00 00 00 .......x........
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867c0 00 00 00 00 0000 00 01 - 00 00 00 01 10 3e df b0 .............>..
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867d0 00 00 00 01 1046 49 90 - 00 00 00 01 10 46 4a 30 .....FI......FJ0
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867e0 00 00 00 01 1046 4c 90 - 6e 78 6a 63 64 62 31 00 .....FL.nxjcdb1.
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867f0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086800 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086810 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086820 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086830 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086840 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086850 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086860 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086870 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086880 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086890 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868a0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868b0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868c0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868e0 00 00 00 00 0000 00 00 - 6e 78 6a 63 64 62 31 2d ........nxjcdb1-
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100868f0 70 72 69 00 0000 00 00 - 00 00 00 00 00 00 00 00 pri.............
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086900 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086910 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086920 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086930 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086940 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086950 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086960 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086970 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086980 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086990 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869a0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869b0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869c0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869e0 00 00 00 00 0000 00 00 - 2f 6f 72 61 63 6c 65 2f ......../oracle/
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869f0 70 72 6f 64 7563 74 2f - 31 30 2e 32 2e 30 2f 63 product/10.2.0/c
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a00 72 73 5f 31 0000 00 00 - 00 00 00 00 00 00 00 00 rs_1............
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a10 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086a20 00 00 00 00 00 00 00 00 - 00 0000 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a30 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086a40 00 00 00 00 00 00 00 00 - 00 0000 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a50 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086a60 00 00 00 00 00 00 00 00 - 00 0000 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a70 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086a80 00 00 00 00 00 00 00 00 - 00 0000 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a90 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086aa0 00 00 00 00 00 00 00 00 -00 00 00 00 00 00 00 00 ................
/dev/rlvjc_voting
3. 日志报错,节点1的crs服务无法启动。
根据报错信息,初步判定是因为节点1无法voting disk造成 OCSSD无法启动。
4. 问题定位
发现/etc/oracle/scls_scr/ballontt(主机名)/oracle路径下缺少cssfatal文件,此文件里只有“enable”一个单词
~cat cssfatal
Enable
5. 问题解决
手动添加该文件
~vi cssfatal
Enable
6. 成功启动节点1的CRS