【RAC】cssfatal缺少导致的节点1无法启动
- 高洛峰原创
- 2016-11-24 11:26:071372浏览
环境：AIX5.3+10.2.0.5 RAC
情境描述：RAC在关闭后重新启动时，一节点无法启动，二节点正常启动
排错过程：
1. 尝试启动节点1的crs服务
root# ./init.crs start crs
2. 监控启动过程中，crs的日志
OCSSD.log日志：
[ CSSD]2014-01-16 09:27:54.730 >USER: Copyright 2014, Oracle version 10.2.0.5.0
[ CSSD]2014-01-16 09:27:54.730 >USER: Starting CSS daemon on node nxjcdb1, number1, in cluster crs_dljc
[ clsdmt]Listening to(ADDRESS=(PROTOCOL=ipc)(KEY=nxjcdb1DBG_CSSD))
[ CSSD]2014-01-16 09:27:54.790 [1]>TRACE: clssscmain: RT queuesetting: ON
[ CSSD]2014-01-16 09:27:55.081 [1]>TRACE: clssscmain: local-only setto false
[ CSSD]2014-01-16 09:27:55.349 [1]>TRACE: clssnmReadNodeInfo: addednode 1 (nxjcdb1) to cluster
[ CSSD]2014-01-16 09:27:55.672 [1]>TRACE: clssnmReadNodeInfo: addednode 2 (nxjcdb2) to cluster
[ CSSD]2014-01-16 09:27:55.673 [1]>TRACE: clssnmInitNMInfo:Initialized with unique 1389835674
[ CSSD]2014-01-16 09:27:55.704 [1]>TRACE: clssNMInitialize:Initializing with OCR id (1516675067)
[ CSSD]2014-01-16 09:27:55.705 [1029] >TRACE: clssnm_skgxninit: HACMP clusterware detected
[ CSSD]2014-01-16 09:27:56.822 [1]>TRACE: clssnmNMInitialize:misscount set to (30)
[ CSSD]2014-01-16 09:27:56.900 [1]>TRACE: clssnmStartNM: reboottimeset to (3) sec
[ CSSD]2014-01-16 09:27:56.900 [1]>TRACE: clssnmNMInitialize: Networkheartbeat thresholds are: impending reconfig 15000 ms, reconfig start(misscount) 30000 ms
[ CSSD]2014-01-16 09:27:57.108 [1]>TRACE: clssnmDiskStateChange: statefrom 1 to 2 disk (0//dev/rlvjc_voting)
[ CSSD]2014-01-16 09:27:57.108 [1030]>TRACE: clssnmvDPT: spawned for disk0 (/dev/rlvjc_voting)
[ CSSD]2014-01-16 09:27:57.146 [1030]>TRACE: clssnmvDiskOpen: Overwrotekill block for voting disk /dev/rlvjc_voting
[ CSSD]2014-01-16 09:27:59.163 [1030]>TRACE: clssnmDiskStateChange: statefrom 2 to 4 disk (0//dev/rlvjc_voting)
[ CSSD]2014-01-16 09:27:59.164 [1]>ERROR: Internal Error Information:
Category: 1234
Operation: scls_scr_setval
Location: open
Other: cant open file
Dep: 2
[ CSSD]2014-01-16 09:27:59.164 [1]>ERROR: clssscSclsFatal: failure 8reading fatal mode
[ CSSD]2014-01-16 09:27:59.164 [1]>ERROR: ###################################
[ CSSD]2014-01-16 09:27:59.164 [1]>ERROR: clssscExit: CSSD abortingfrom thread Main
[ CSSD]2014-01-16 09:27:59.164 [1]>ERROR: ###################################
→根据报错信息，初步判定是因为节点1无法voting disk造成 OCSSD无法启动。
[ CSSD]--- DUMP GROCK STATE DB ---
[ CSSD]--- END OF GROCK STATE DUMP ---
[ CSSD]2014-01-16 09:27:59.169 [1030]>TRACE: clssnmvReadDskHeartbeat:read ALL for Joining
[ CSSD]2014-01-16 09:27:59.169 [1030]>TRACE: clssnmvReadDskHeartbeat:node(2) is down. rcfg(2) wrtcnt(126947) LATS(1038806686) Disk lastSeqNo(126947)
[ CSSD]------- Begin Dump -------
[ CSSD]
[ CSSD]
[ CSSD]
[ CSSD]
[ CSSD]
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100863c0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100863d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100863e0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100863f0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086400 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086410 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086420 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086430 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086440 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086450 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086460 00 00 00 01 1008 61 98 - 00 00 00 01 10 c6 0b c0 ......a.........
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086470 00 00 00 00 00 0000 01 - 00 00 00 00 00 02 00 03 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086480 00 00 00 01 1096 4a b0 - 00 00 00 00 00 00 00 00 ......J.........
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086490 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864a0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864b0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 01 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864c0 00 00 00 00 0000 00 05 - 00 00 00 00 00 00 00 fa ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864e0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864f0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086500 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086510 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086520 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086530 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086540 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086550 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086560 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086570 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086580 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086590 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865a0 00 00 00 00 0000 00 00 - 00 00 00 0e 00 00 00 24 ...............$
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865b0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865c0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865e0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865f0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086600 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086610 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086620 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086630 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086640 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086650 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086660 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086670 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086680 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086690 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866a0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866b0 00 00 10 00 0000 00 97 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866c0 00 00 00 01 105d f6 10 - 00 00 00 01 10 95 ca 50 .....].........P
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866d0 00 00 00 01 1096 0a 70 - 00 00 00 3c 00 00 00 00 .......p...<....
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866e0 00 00 00 01 1096 2a 90 - 00 00 00 01 00 00 00 01 ......*.........
[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866f0 00 00 00 28 0000 00 00 - 00 00 00 01 10 00 16 08 ...(............
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086700 00 00 00 00 0000 00 00 - 00 00 00 01 10 4c 1a 90 .............L..
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086710 00 00 00 01 104c 1a 50 - 00 00 00 01 10 08 67 18 .....L.P......g.
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086720 00 00 00 01 1008 67 18 - 00 00 00 00 00 00 00 00 ......g.........
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086730 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086740 00 00 00 01 10cf 76 b0 - 00 00 00 00 00 00 00 00 ......v.........
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086750 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086760 00 00 00 01 10cf 76 b0 - 00 00 00 00 00 00 00 00 ......v.........
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086770 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086780 00 00 01 00 0000 00 00 - 00 00 00 01 10 bd f7 90 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086790 00 00 00 01 104c 17 50 - 00 00 00 00 00 00 00 00 .....L.P........
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867a0 00 00 00 00 0000 00 00 - 00 00 00 03 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867b0 00 00 00 01 1000 09 78 - 00 00 00 01 00 00 00 00 .......x........
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867c0 00 00 00 00 0000 00 01 - 00 00 00 01 10 3e df b0 .............>..
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867d0 00 00 00 01 1046 49 90 - 00 00 00 01 10 46 4a 30 .....FI......FJ0
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867e0 00 00 00 01 1046 4c 90 - 6e 78 6a 63 64 62 31 00 .....FL.nxjcdb1.
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867f0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086800 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086810 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086820 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086830 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086840 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086850 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086860 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086870 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086880 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086890 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868a0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868b0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868c0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868e0 00 00 00 00 0000 00 00 - 6e 78 6a 63 64 62 31 2d ........nxjcdb1-
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100868f0 70 72 69 00 0000 00 00 - 00 00 00 00 00 00 00 00 pri.............
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086900 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086910 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086920 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086930 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086940 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086950 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086960 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086970 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086980 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086990 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869a0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869b0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869c0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869e0 00 00 00 00 0000 00 00 - 2f 6f 72 61 63 6c 65 2f ......../oracle/
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869f0 70 72 6f 64 7563 74 2f - 31 30 2e 32 2e 30 2f 63 product/10.2.0/c
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a00 72 73 5f 31 0000 00 00 - 00 00 00 00 00 00 00 00 rs_1............
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a10 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086a20 00 00 00 00 00 00 00 00 - 00 0000 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a30 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086a40 00 00 00 00 00 00 00 00 - 00 0000 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a50 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086a60 00 00 00 00 00 00 00 00 - 00 0000 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a70 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086a80 00 00 00 00 00 00 00 00 - 00 0000 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a90 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................
[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086aa0 00 00 00 00 00 00 00 00 -00 00 00 00 00 00 00 00 ................
/dev/rlvjc_voting
3. 日志报错，节点1的crs服务无法启动。
根据报错信息，初步判定是因为节点1无法voting disk造成 OCSSD无法启动。
4. 问题定位
发现/etc/oracle/scls_scr/ballontt(主机名)/oracle路径下缺少cssfatal文件，此文件里只有“enable”一个单词
~cat cssfatal
Enable
5. 问题解决
手动添加该文件
~vi cssfatal
Enable
6. 成功启动节点1的CRS