一个RAC数据库，意外DOWN机后，第一个节点正常启动，但是第二个节点却无法启动ASM和CRS资源。
1、现象描述
一个RAC数据库，意外DOWN机后，第一个节点正常启动，但是第二个节点却无法启动ASM和CRS资源。
2、分析原因
由于ASM磁盘组无法启动，查看ASM日志发现如下信息：
MMNL started with pid=21,OS id=14028
lmon registered with NM -instance number 2 (internal mem no 1)
Tue Nov 18 14:48:50 2014
PMON (ospid:13986): terminating the instance due to error 481
Tue Nov 18 14:48:50 2014
System state dumprequested by (instance=2, osid=13986 (PMON)), summary=[abnormal instancetermination].
System State dumped totrace file /Oracle/product/grid/diag/asm/+asm/+ASM2/trace/+ASM2_diag_13996.trc
Tue Nov 18 14:48:50 2014
ORA-1092 :opitsk aborting process
Dumping diagnostic data indirectory=[cdmp_20141118144850], requested by (instance=2, osid=13986 (PMON)),summary=[abnormal instance termination].
Tue Nov 18 14:48:50 2014
ORA-1092 : opitsk abortingprocess
Instance terminated byPMON, pid = 13986
在Metalink网站上，发现一篇文档ASM onNon First Node (Second or Other Node) Fails to Come up With: PMON (ospid:nnnn): terminating the instance due to error 481 [ID 1383737.1]。依据该文档得知，ASM的alert日志错误信息中“PMON(ospid: 9946): terminating the instance due to error 481”文档的标题是一致。根据文档的内容进一步检查相关日志及配置信息，发现ASM TRACE也如文档一致。如下：
*** 2014-11-18 14:48:17.092
Reconfiguration completes [incarn=42]
kjzdattdlm: Can not attach to DLM (LMONup=[TRUE], DB mounted=[FALSE]).
通过分析集群日志发现如下信息：
2014-11-18 14:44:45.767
[/oracle/product/11.2.0/grid/bin/orarootagent.bin(12690)]CRS-5018:(:CLSN00037:)Removed unused HAIP route: 169.254.95.0/ 255.255.255.0 / 0.0.0.0 / usb0
说明ORACLE认为usb0（主机管理口网卡，默认开启）占用了HAIP（ORACLE 11G新特性，内部的高可用私有IP，通过HAIP替换两个节点的私有IP,来进行内部通讯）。
通过分析主机日志发现如下信息：
Nov 18 14:02:11 XXXdb2 dhclient:DHCPREQUEST on usb0 to 255.255.255.255 port 67
Nov 18 14:02:12 XXXdb2 dhclient:DHCPACK from 169.254.95.118
Nov 18 14:02:12 XXXdb2 dhclient: boundto 169.254.95.120 -- renewal in 234 seconds.
说明主机网卡USB0在动态获取169.254.XX.XX网段的IP地址。
IBM的PC服务器使用USB0做为管理网络的特性。没有连接USB0网卡的时候会不停向DHCP申请IP，如果没有发现DHCP时就会默认分配一个169.254.xxx.xxx的IP地址会和ORACLE的HAIP产生冲突,造成路由信息丢失
通过各种日志信息与文档中的信息的对比，得知此次的故障现象与文档中的故障现象是一致的。
3、解决方法
根据文档ID 1383737.1提供的内容，使用ifdown usb0关闭两个节点的usb0网卡后，在缺失路由信息的一节点动态添加路由信息
#route add -net 169.254.0.0 netmask 255.255.0.0 dev eth2
增加路由之后，在数据库的第二节点，通过命令
#su - grid
$GRID_HOME/bin/crsctl start res ora.crsd -init
正常启动crsd资源后，asm和crsd资源全部正常启动。
4、故障总结
IBM 的x3850 x5系列的PC Server存在USB开启dhcp功能，从而导致usb网卡可能占用HAIP的缺陷，，生产环境中的此类机器上运行的RAC数据库环境，需要关闭USB0的自动获取dhcp功能，给USB0配置静态IP。
5 、相关参考
ASM on Non First Node (Second or Other Node) Fails to Come up With:PMON (ospid: nnnn): terminating the instance due to error 481 [ID 1383737.1]
如何修改ASM的sys密码
如何将ASM中的数据文件复制到操作系统中
Oracle 11g RAC ASM磁盘全部丢失后的恢复
Oracle 11g从入门到精通 PDF+光盘源代码
RHEL6 ASM方式安装Oracle 11g R2
Oracle 10g 手工创建ASM数据库
Ubuntu 12.04(amd64)安装完Oracle 11gR2后各种问题解决方法
如何修改Oracle 10g ASM的sys密码
本文永久更新链接地址：
Explain the ACID properties (Atomicity, Consistency, Isolation, Durability).Apr 16, 2025 am 12:20 AMACID attributes include atomicity, consistency, isolation and durability, and are the cornerstone of database design. 1. Atomicity ensures that the transaction is either completely successful or completely failed. 2. Consistency ensures that the database remains consistent before and after a transaction. 3. Isolation ensures that transactions do not interfere with each other. 4. Persistence ensures that data is permanently saved after transaction submission.
MySQL: Database Management System vs. Programming LanguageApr 16, 2025 am 12:19 AMMySQL is not only a database management system (DBMS) but also closely related to programming languages. 1) As a DBMS, MySQL is used to store, organize and retrieve data, and optimizing indexes can improve query performance. 2) Combining SQL with programming languages, embedded in Python, using ORM tools such as SQLAlchemy can simplify operations. 3) Performance optimization includes indexing, querying, caching, library and table division and transaction management.
MySQL: Managing Data with SQL CommandsApr 16, 2025 am 12:19 AMMySQL uses SQL commands to manage data. 1. Basic commands include SELECT, INSERT, UPDATE and DELETE. 2. Advanced usage involves JOIN, subquery and aggregate functions. 3. Common errors include syntax, logic and performance issues. 4. Optimization tips include using indexes, avoiding SELECT* and using LIMIT.
MySQL's Purpose: Storing and Managing Data EffectivelyApr 16, 2025 am 12:16 AMMySQL is an efficient relational database management system suitable for storing and managing data. Its advantages include high-performance queries, flexible transaction processing and rich data types. In practical applications, MySQL is often used in e-commerce platforms, social networks and content management systems, but attention should be paid to performance optimization, data security and scalability.
SQL and MySQL: Understanding the RelationshipApr 16, 2025 am 12:14 AMThe relationship between SQL and MySQL is the relationship between standard languages and specific implementations. 1.SQL is a standard language used to manage and operate relational databases, allowing data addition, deletion, modification and query. 2.MySQL is a specific database management system that uses SQL as its operating language and provides efficient data storage and management.
Explain the role of InnoDB redo logs and undo logs.Apr 15, 2025 am 12:16 AMInnoDB uses redologs and undologs to ensure data consistency and reliability. 1.redologs record data page modification to ensure crash recovery and transaction persistence. 2.undologs records the original data value and supports transaction rollback and MVCC.
What are the key metrics to look for in an EXPLAIN output (type, key, rows, Extra)?Apr 15, 2025 am 12:15 AMKey metrics for EXPLAIN commands include type, key, rows, and Extra. 1) The type reflects the access type of the query. The higher the value, the higher the efficiency, such as const is better than ALL. 2) The key displays the index used, and NULL indicates no index. 3) rows estimates the number of scanned rows, affecting query performance. 4) Extra provides additional information, such as Usingfilesort prompts that it needs to be optimized.
What is the Using temporary status in EXPLAIN and how to avoid it?Apr 15, 2025 am 12:14 AMUsingtemporary indicates that the need to create temporary tables in MySQL queries, which are commonly found in ORDERBY using DISTINCT, GROUPBY, or non-indexed columns. You can avoid the occurrence of indexes and rewrite queries and improve query performance. Specifically, when Usingtemporary appears in EXPLAIN output, it means that MySQL needs to create temporary tables to handle queries. This usually occurs when: 1) deduplication or grouping when using DISTINCT or GROUPBY; 2) sort when ORDERBY contains non-index columns; 3) use complex subquery or join operations. Optimization methods include: 1) ORDERBY and GROUPB
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
Atom editor mac version download
The most popular open source editor
MinGW - Minimalist GNU for Windows
This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.
EditPlus Chinese cracked version
Small size, syntax highlighting, does not support code prompt function
Dreamweaver Mac version
Visual web development tools
Notepad++7.3.1
Easy-to-use and free code editor
