一次RAC VIP漂移的结果诊断及修复-Mysql Tutorial-php.cn

Home

Database

Mysql Tutorial

一次RAC VIP漂移的结果诊断及修复

WBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWB

Jun 07, 2016 pm 04:00 PM

背景概述客户的10G数据库VIP出现宕，引起VIP负载到另一个节点事件支持细节 04:29:56.378 一号机器VIP 出现 went OFFLINE unexp

背景概述

客户的10G数据库VIP出现宕，引起VIP负载到另一个节点

事件支持细节

04:29:56.378 一号机器VIP 出现 went OFFLINE unexpectedly，当天出现这个VIP漂移的故障后为检查VIP宕掉的原因，

对VIP资源启动DEBUG 5模式：./crsctl debug log res "orahostname1.vip:5"

04:38:36.047 一号节点VIP 出现 went OFFLINE unexpectedly。

根据ora.hostname.vip.log日志显示，出现VIP宕原因基本可以确定为公网IP与缺省网管通讯不畅引起。

根据Oracle管方建议，调整racgvip程序中的参数从 FAIL_WHEN_DEFAULTGW_NO_FOUND=1 修改成

FAIL_WHEN_DEFAULTGW_NO_FOUND=0

但是调整完后故障依旧

04:17:37.822: [ CRSRES][11025]32ora.hostname1.vip on hostname1 went OFFLINE unexpectedly

为明确原因，再次收集ora.hostname1.vip.log及racgvip 信息进行分析

分析结果如下：

在racgvip程序中，有如下代码

# Check the status of the interface thro' pinging gateway

if [ -n "$DEFAULTGW" ]

then

_RET=1

# get base IP address of the interface

tmpIP=`$LSATTR -El ${_IF} -a netaddr | $AWK '{print $2}'`

# get RX packets numbers (bug8341569,9157855->bug9743421)

_O1=`$NETSTAT -n -I $_IF | $AWK "{ if (/^$_IF/) {print \\$(NF-4); exit}}"`

x=$CHECK_TIMES

while [ $x -gt 0 ]

if [ -n "$tmpIP" ]

then

logx "About to execute command: $PING -S $tmpIP $PING_TIMEOUT $DEFAULTGW"

$PING -S $tmpIP $PING_TIMEOUT $DEFAULTGW > /dev/null 2>&1

else

logx "About to execute command: $PING $PING_TIMEOUT $DEFAULTGW"

$PING $PING_TIMEOUT $DEFAULTGW > /dev/null 2>&1

_O2=`$NETSTAT -n -I $_IF | $AWK "{ if (/^$_IF/) {print \\$(NF-4); exit}}"`

if [ "$_O1" != "$_O2" ]

then

# RX packets numbers changed

_RET=0

break

$SLEEP 1

x=`$EXPR $x - 1`

done

if [ $_RET -ne 0 ]

then

logx "IsIfAlive: RX packets checked if=$_IF failed"

else

logx "IsIfAlive: RX packets checked if=$_IF OK"

else

logx "IsIfAlive: Default gateway is not defined (host=$HOSTNAME)"

if [ $FAIL_WHEN_DEFAULTGW_NO_FOUND -eq 1 ]

then

_RET=1

else

_RET=0

从源码我们可以看到检查缺省网关的处理逻辑

1、如果检测到缺省网关存在执行网管检查逻辑

2、_01收集网卡网络包量

3、$PING -S $tmpIP $PING_TIMEOUT $DEFAULTGW ping网管

4、_02再次收集网卡网络包量

5、如果_01网卡网络包量与 _02网卡网络包量不相同，表明网卡与缺省网卡之间通讯正常 _RET 返回编码为0

6、如果_01网卡网络包量与 _02网卡网络包量相同，_RET 返回编码没指定，缺省返回1，同时打印日志logx "IsIfAlive: RX packets checked if=$_IF failed"，即判断网卡失败。

FAIL_WHEN_DEFAULTGW_NO_FOUND参数从1修改成0，是为了跳过网关ping检测，而从源码中我们可以看到，FAIL_WHEN_DEFAULTGW_NO_FOUND参数只有在网卡参数$DEFAULTGW为空才生效，即主机上没有配置网关并且参数FAIL_WHEN_DEFAULTGW_NO_FOUND配置为非1时返回码RET为0。

由于我们的环境中DEFAULTGW能获取成功及DEFAULTGW非空，导致程序没有进入FAIL_WHEN_DEFAULTGW_NO_FOUND判断是否为1的处理流程。

故障期间DEBUG错误信息如下：

2013-11-06 04:17:37.776: [ RACG][1] [18219068][1][ora.s9lp1.vip]: Wed Nov 6 04:17:33 CST 2013 [ 6422696 ] checkIf: start for if=en5

Wed Nov 6 04:17:33 CST 2013 [ 6422696 ] IsIfAlive: start for if=en5

Wed Nov 6 04:17:33 CST 2013 [ 6422696 ] defaultgw: started

2013-11-06 04:17:37.776: [ RACG][1] [18219068][1][ora.s9lp1.vip]: Wed Nov 6 04:17:33 CST 2013 [ 6422696 ] defaultgw: completed with 10.0.241.254 （网关获取成功，网关为10.0.241.254）

Wed Nov 6 04:17:33 CST 2013 [ 6422696 ] About to execute command: /usr/sbin/ping -S 10.0.241.150 -c 1 -w 1 10.0.241.254

2013-11-06 04:17:37.777: [ RACG][1] [18219068][1][ora.s9lp1.vip]: Wed Nov 6 04:17:35 CST 2013 [ 6422696 ] About to execute command: /usr/sbin/ping -S 10.0.241.150 -c 1 -w 1 10.0.241.254 （PING 网关）

Wed Nov 6 04:17:37 CST 2013 [ 6422696 ] IsIfAlive: RX packets checked if=en5 failed（由于检查到网卡en5在2秒中内网卡流量包未方式变化，判断为en5失败）

1、故障每次发生都在凌晨04左右，时间如下：

2013-10-28 04:29:56

2013-11-01 04:38:36

2013-11-06 04:17:37
2、从源码上分析，发生故障期间网卡en5连续1秒的网络包未变化

可能的原因：

ping -S 10.0.241.150 -c 1 -w 1 10.0.241.254

Oracle检测网管时，由于当时网络质量不好导致ping不能在1秒钟内返回结果。

引起网卡en5 ping前 ping后没有网络包发生变化。

根据以上分析我们建议：

1、修改racgvip源码跳过网管检测

修改前：

# Check the status of the interface thro' pinging gateway

if [ -n "$DEFAULTGW" ]

修改后：

# Check the status of the interface thro' pinging gateway
if [ -n "$DEFAULTGW" -a $FAIL_WHEN_DEFAULTGW_NO_FOUND -eq 1 ]

查阅oracle11.2.0.3版本的 RACGVIP代码，同样以次修改

以下为Oracle11G的racgvip代码

if [ -n "$DEFAULTGW" -a $FAIL_WHEN_DEFAULTGW_NOT_FOUND -eq 1 ]

then

_RET=1

# get RX packets numbers

_O1=`$IFCONFIG $_IF | $AWK '{ if (/RX packets:/) { sub("packets:", "", $2); print $2}}'`

x=$CHECK_TIMES

while [ $x -gt 0 ]

logx "About to execute $PING -r -I $_IF $DEFAULTGW $PING_TIMEOUT"

$PING -r -I $_IF $DEFAULTGW $PING_TIMEOUT > /dev/null 2>&1

rc=$?

if [ $rc -eq 0 ]

then

_RET=0

break

else

echo "ping to $DEFAULTGW via $_IF failed, rc = $rc (host=$HOSTNAME)"

x=$(($x-1))

done

结论及解决方案

修改racgvip代码

修改完成后，，需要观察ora.s9lp1.vip.log里出现如下信息：

IsIfAlive: Default gateway is not defined (host=$HOSTNAME)

表明修改失效

本文永久更新链接地址：

Statement

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Explain the role of InnoDB redo logs and undo logs.Apr 15, 2025 am 12:16 AM

InnoDB uses redologs and undologs to ensure data consistency and reliability. 1.redologs record data page modification to ensure crash recovery and transaction persistence. 2.undologs records the original data value and supports transaction rollback and MVCC.

What are the key metrics to look for in an EXPLAIN output (type, key, rows, Extra)?Apr 15, 2025 am 12:15 AM

Key metrics for EXPLAIN commands include type, key, rows, and Extra. 1) The type reflects the access type of the query. The higher the value, the higher the efficiency, such as const is better than ALL. 2) The key displays the index used, and NULL indicates no index. 3) rows estimates the number of scanned rows, affecting query performance. 4) Extra provides additional information, such as Usingfilesort prompts that it needs to be optimized.

What is the Using temporary status in EXPLAIN and how to avoid it?Apr 15, 2025 am 12:14 AM

Usingtemporary indicates that the need to create temporary tables in MySQL queries, which are commonly found in ORDERBY using DISTINCT, GROUPBY, or non-indexed columns. You can avoid the occurrence of indexes and rewrite queries and improve query performance. Specifically, when Usingtemporary appears in EXPLAIN output, it means that MySQL needs to create temporary tables to handle queries. This usually occurs when: 1) deduplication or grouping when using DISTINCT or GROUPBY; 2) sort when ORDERBY contains non-index columns; 3) use complex subquery or join operations. Optimization methods include: 1) ORDERBY and GROUPB

Describe the different SQL transaction isolation levels (Read Uncommitted, Read Committed, Repeatable Read, Serializable) and their implications in MySQL/InnoDB.Apr 15, 2025 am 12:11 AM

MySQL/InnoDB supports four transaction isolation levels: ReadUncommitted, ReadCommitted, RepeatableRead and Serializable. 1.ReadUncommitted allows reading of uncommitted data, which may cause dirty reading. 2. ReadCommitted avoids dirty reading, but non-repeatable reading may occur. 3.RepeatableRead is the default level, avoiding dirty reading and non-repeatable reading, but phantom reading may occur. 4. Serializable avoids all concurrency problems but reduces concurrency. Choosing the appropriate isolation level requires balancing data consistency and performance requirements.

MySQL vs. Other Databases: Comparing the OptionsApr 15, 2025 am 12:08 AM

MySQL is suitable for web applications and content management systems and is popular for its open source, high performance and ease of use. 1) Compared with PostgreSQL, MySQL performs better in simple queries and high concurrent read operations. 2) Compared with Oracle, MySQL is more popular among small and medium-sized enterprises because of its open source and low cost. 3) Compared with Microsoft SQL Server, MySQL is more suitable for cross-platform applications. 4) Unlike MongoDB, MySQL is more suitable for structured data and transaction processing.

How does MySQL index cardinality affect query performance?Apr 14, 2025 am 12:18 AM

MySQL index cardinality has a significant impact on query performance: 1. High cardinality index can more effectively narrow the data range and improve query efficiency; 2. Low cardinality index may lead to full table scanning and reduce query performance; 3. In joint index, high cardinality sequences should be placed in front to optimize query.

MySQL: Resources and Tutorials for New UsersApr 14, 2025 am 12:16 AM

The MySQL learning path includes basic knowledge, core concepts, usage examples, and optimization techniques. 1) Understand basic concepts such as tables, rows, columns, and SQL queries. 2) Learn the definition, working principles and advantages of MySQL. 3) Master basic CRUD operations and advanced usage, such as indexes and stored procedures. 4) Familiar with common error debugging and performance optimization suggestions, such as rational use of indexes and optimization queries. Through these steps, you will have a full grasp of the use and optimization of MySQL.

Real-World MySQL: Examples and Use CasesApr 14, 2025 am 12:15 AM

MySQL's real-world applications include basic database design and complex query optimization. 1) Basic usage: used to store and manage user data, such as inserting, querying, updating and deleting user information. 2) Advanced usage: Handle complex business logic, such as order and inventory management of e-commerce platforms. 3) Performance optimization: Improve performance by rationally using indexes, partition tables and query caches.

See all articles