ASM DISK Group加载ORA-15183错误一例

进入Oracle 11gR2，ASM（Automatic Storage Management）从Database组件中剥离出来，作为独立组件Component进入Grid管理范畴。

进入Oracle 11gR2，ASM（Automatic Storage Management）从Database组件中剥离出来，作为独立组件Component进入Grid管理范畴。

本篇主要介绍笔者遇到的一个数据库启动加载过程中出现的问题。同官方MOS推荐的策略相比，有一些不同之处。记录下来，留待需要的朋友待查使用。

1、问题说明

笔者环境是Oracle 单实例+Grid Infrastructure，版本号为11.2.0.4。由于安全原因，从MOS上下载了最新的安全补丁和升级补丁。升级之后的版本为11.2.0.4.6。

但是，在升级最后步骤——执行SQL脚本环节，出现了一些问题。

SQL*Plus: Release 11.2.0.4.0 Production on Mon May 25 16:08:57 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

SQL> conn / as sysdba

Connected to an idle instance.

SQL> startup

ORACLE instance started.

Total System Global Area 2087780352 bytes

Fixed Size                  2254824 bytes

Variable Size            553650200 bytes

Database Buffers        1526726656 bytes

Redo Buffers                5148672 bytes

ORA-00205: error in identifying control file, check alert log for more info

从提示信息角度看，Oracle在经历启动nomount阶段之后，在定位control file的过程中出现了问题。

老实说，虽然是测试环境，但是笔者还是比较惊慌的。于是尝试使用srvctl集群件启动策略。

[grid@NCR-Standby-Asm ~]$ srvctl start database -d sicsstb

PRCC-1014 : sicsstb was already running

PRCR-1004 : Resource ora.sicsstb.db is already running

PRCR-1079 : Failed to start resource ora.sicsstb.db

CRS-5702: Resource 'ora.sicsstb.db' is already running on 'ncr-standby-asm'

2、问题分析

首先确认系统是否可以使用srvctl启动，判断一下GI上面各种资源resource状态。

[grid@NCR-Standby-Asm ~]$ srvctl stop database -d sicsstb

[grid@NCR-Standby-Asm ~]$ srvctl status asm

ASM is running on ncr-standby-asm

[grid@NCR-Standby-Asm ~]$ crsctl stat res -t -init

--------------------------------------------------------------------------------

NAME          TARGET  STATE        SERVER                  STATE_DETAILS       

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA.dg

              ONLINE  ONLINE      ncr-standby-asm                             

ora.LISTENER.lsnr

              ONLINE  ONLINE      ncr-standby-asm                             

ora.RECO.dg

              ONLINE  ONLINE      ncr-standby-asm                             

ora.asm

              ONLINE  ONLINE      ncr-standby-asm          Started             

ora.ons

              OFFLINE OFFLINE      ncr-standby-asm                             

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.cssd

      1        ONLINE  ONLINE      ncr-standby-asm                             

ora.diskmon

      1        OFFLINE OFFLINE                                                   

ora.evmd

      1        ONLINE  ONLINE      ncr-standby-asm                             

ora.sicsstb.db

      1        OFFLINE OFFLINE                              Instance Shutdown   

[grid@NCR-Standby-Asm ~]$ srvctl start database -d sicsstb

[grid@NCR-Standby-Asm ~]$ 

[oracle@NCR-Standby-Asm ~]$ cd $ORACLE_HOME/rdbms/admin

[oracle@NCR-Standby-Asm admin]$ sqlplus /nolog

SQL*Plus: Release 11.2.0.4.0 Production on Mon May 25 16:14:00 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

SQL> conn / as sysdba

Connected.

SQL> select open_mode from v$database;

OPEN_MODE

--------------------

READ WRITE

笔者猜测，这个故障和ASM相关。按照逐步抽丝剥茧的思路，先从数据库日志入手（找到失败启动的那次动作）。

Mon May 25 16:09:28 2015

MMON started with pid=17, OS id=4151 

Mon May 25 16:09:28 2015

MMNL started with pid=18, OS id=4153 

starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...

starting up 1 shared server(s) ...

NOTE: initiating MARK startup 

Starting background process MARK

ORACLE_BASE from environment = /u02/app/oracle

Mon May 25 16:09:28 2015

MARK started with pid=21, OS id=4161 

NOTE: MARK has subscribed 

Mon May 25 16:09:28 2015

ALTER DATABASE  MOUNT

Mon May 25 16:09:28 2015

ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521))' SCOPE=MEMORY SID='sicsstb';

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so 

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

SUCCESS: diskgroup DATA was dismounted

ERROR: diskgroup DATA was not mounted

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so 

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

SUCCESS: diskgroup RECO was dismounted

ERROR: diskgroup RECO was not mounted

ORA-00210: cannot open the specified control file

ORA-00202: control file: '+RECO/sicsstb/controlfile/current.256.878897845'

ORA-17503: ksfdopn:2 Failed to open file +RECO/sicsstb/controlfile/current.256.878897845

ORA-15001: diskgroup "RECO" does not exist or is not mounted

ORA-15040: diskgroup is incomplete

ORA-15040: diskgroup is incomplete

ORA-00210: cannot open the specified control file

ORA-00202: control file: '+DATA/sicsstb/controlfile/current.260.878897845'

ORA-17503: ksfdopn:2 Failed to open file +DATA/sicsstb/controlfile/current.260.878897845

ORA-15001: diskgroup "DATA" does not exist or is not mounted

ORA-15040: diskgroup is incomplete

ORA-15040: diskgroup is incomplete

ORA-15040: diskgroup is incomplete

ORA-205 signalled during: ALTER DATABASE  MOUNT...

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so 

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

Mon May 25 16:09:31 2015

SUCCESS: diskgroup DATA was dismounted

ERROR: diskgroup DATA was not mounted

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so 

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

SUCCESS: diskgroup RECO was dismounted

ERROR: diskgroup RECO was not mounted

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

从提示信息看，Oracle在mount阶段时候，利用spfile中指定的control file位置去访问+DATA和+RECO磁盘组，，但是两个磁盘组没有mount，所以才开始报错。

参数中，control file以镜像冗余方式存在在ASM Diskgroup中。

SQL> show parameter spfile

NAME                                TYPE        VALUE

------------------------------------ ----------- ------------------------------

spfile  string      +DATA/sicsstb/spfilesicsstb.ora

SQL> show parameter control

NAME                                TYPE        VALUE

------------------------------------ ----------- ------------------------------

control_file_record_keep_time        integer    7

control_files                        string      +DATA/sicsstb/controlfile/curr