实例宕机引发的ORA-00240错误

事故第二天进行了数据库的alert log分析，从日志中可以看到数据库在实例NODE2发生宕机后，RAC已经做出了实例切换步骤，但在切换的

1.环境描述

OS：AIX6.1

Oracle ：11.2.0.3.0 RAC

2.事故发生

数据库NODE 2所在小型机发生宕机事故，，本应正常切换至NODE1，但切换失败，重启系统得以解决。

3.事故分析

事故第二天进行了数据库的alert log分析，从日志中可以看到数据库在实例NODE2发生宕机后，RAC已经做出了实例切换步骤，但在切换的过程中遭遇了ORA-00240、ORA-29770错误，导致当时数据库没有切换成功。下面是日志的详细分析。

数据库在当时经历了大致以下几个重要步骤：

1.  Beginning instance recovery of 1 threads

数据库开始在本地恢复宕机的实例

2.  Started redo application at

Thread 2: logseq 4556, block 368380

    数据库开始从在线redo日志序号为4556的日志恢复

3.  Completed instance recovery at

Thread 2: logseq 4556, block 376983, scn 3502123313

    数据库redo日志4556已经恢复成功。

4.  Redo thread 2 internally disabled at seq 4557 (SMON)

当数据库准备恢复4557的日志时，遭遇失败。

5  ORA-00240: control file enqueue held for more than 120 seconds

日志中开始出现ora-00240错误，提示控制文件被持有超过120秒。

6  ORA-29770: global enqueue process DIA0 (OSID 12517556) is hung for more than 300 seconds

Incident details in:

紧接着出现ora29770错误，数据库进程DIA0 hung住超过5分钟。

在当天实例宕机的20分钟左右时间内，当时系统的监控等脚本均无法执行，分析Oracle awr报告得知在当天宕机的时间内，未宕机的NODE1系统CPU资源几乎耗尽。

经过分析与推测，DIA0进程的主要作用是处理数据库死锁、hung住的一个进程，日志中的现象应该是这个进程发现控制文件被持有超过120秒，属于错误行为，进程开始处理这一问题，但当时系统cpu资源耗尽，进程DIA0解决故障超过5分钟，导致出现了ora-29770错误。

所以认定这次实例切换失败的主要原因在于ORA-00240错误。根据日志中给出的trace文件，查看trace文件中的具体报错原因。

从trace日志中分析得知，当时控制文件被持有超过120秒的主要原因是KSV master wait等待，KSV master wait耗费了2分03秒。

4.事故分析结论

根据以上现象及日志体现，从oracle官方metalink中查找资料得知这是一个bug [BUG ID 1308282.1]

下面是官方metalink文档中关于这个bug的解释：

High 'ksv master wait' And 'ASM File Metadata Operation' Waits In Non-Exadata 11g

 

Symptoms

High waits for 'ksv master wait' while doing an ASM file metadata operation were reported when a data migration utility was running.  This wait was also seen for a drop of a tablespace.

The AWR showed the top events were CPU (> 100%), with 'ASM file metadata operation' (7%).

 

Cause

Event 'KSV master wait' indicates the process on the RDBMS side is waiting for a reply from a process on the ASM side.  In 11g, the parameter cell_offload_processing is set to TRUE.  Although that is a parameter is not applicable for non-Exadata databases, it caused ASM to try to deliver smart-scan results.  The issue was reported in Bug 11800170 - ASM IN KSV WAIT AFTER APPLICATION OF 11.2.0.2 GRID PSU.

After applying the workaround for this issue (see Solution below), a drop of a tablespace that used to take 13 minutes took 4 seconds.

 

Solution

The following solutions are available for non-Exadata databases:

For the quickest solution, use the workaround.  The workaround does not negatively impact non-Exadata databases. This parameter is to be set on the database instance.

alter system set cell_offload_processing = false;

Upgrade to 12.1, when available. OR

Apply the 11.2.0.3 patch set  OR

Apply one-off Patch 11800170, if available for your RDBMS and Grid Homes

Note:  At the time this note was written (March 2011), neither 12.1 nor 11.2.0.3 were available.

官方文档中给出的最快解决方式是修改oracle中的一个参数cell_offload_processing，修改为false。