Hadoop集群服务器升级为rhel6内核后,System Cpu占用非常高,有任务运行的时候经常到50%以上。对其中一台机器一天的运行状态采样的数据: idle: 76%?? sys:14%? user: 9% 从采样数据中,可以发现System Cpu比User Cpu还要高,这在Hadoop集群环境中很不寻常。
Hadoop集群服务器升级为rhel6内核后,System Cpu占用非常高,有任务运行的时候经常到50%以上。对其中一台机器一天的运行状态采样的数据:
idle: 76%?? sys:14%? user: 9%
从采样数据中,可以发现System Cpu比User Cpu还要高,这在Hadoop集群环境中很不寻常。
先简单地用strace看了一下占用cpu高的java程序经常去调哪些系统调用,发现sched_yield调用频率非常之高,莫非是锁的问题?分析了下内核中的文档和代码,发现CFS调度下sched_yield的行为与以前的O(1)算法略有出入——CFS下sched_yield返回非常快,对于一些借助sched_yield实现锁的应用来说,开销会很大。内核提供了一个proc参数sched_compat_yield,设置该参数为1,就可以解决这个问题。于是设置了该参数,仍然没有效果,分析代码后,竟然发现sched_compat_yield在rhel6内核中并没有实现,只是留下了一个接口兼容而已。于是乎将upstream中的相关部分的代码port到rhel6的内核中,sched_compact_yield终于能干活了,但出乎意料的是,系统态cpu仍然非常高。
没办法了,上个大招:oprofile,结果如下:
samples???????? %???????? ?symbol name
2822865? ?71.2192?? ?compact_zone
160729??? ?4.0551?????? clear_page_c
156913?? ?? 3.9588???? ?compaction_alloc
47691?????? ?1.2032????? ?copy_user_generic_string
一看到结果,一头雾水。compact_zone为何物?为何cpu占用如此之高?不懂了就看代码。
__alloc_pages_slowpath
__alloc_pages_direct_compact
try_to_compact_pages
compact_zone_order
compact_order
有点头绪了,内核要分配一块高阶物理内存,buddy system中又没有满足条件的,似乎内核要在compact_zone中做些什么事,来满足对高阶物理内存的分配。
下一步,快速验证下是不是compact_zone的问题,修改config文件,去掉CONFIG_COMPACTION,重新编译,换内核,竟然真的OK了 。 那基本断定是compact_zone的问题了,后面就得分析下代码,研究下其中的原理了。
经过几天的艰苦奋战,终于把compaction的基本原理搞明白了。
linux物理内存的管理采用的是经典的伙伴系统,当然也就存在伙伴系统的问题——内存碎片。当然,此处的内存碎片问题并不算大,因为伙伴系统是以页为单位为管理内存的,碎片也是以“页”为单位,4k的物理内存还算不上是“碎片”。对于用户态的程序,几乎不需要超过4k的连续空间。但是对内核来说,碎片永远都不是好东西。某些硬件相关的操作会需要连续的物理内存,如果无法满足,内核就只能panic。
另外,引入compaction的另一个重要因素就是使用THP(Transparent hugepages)。4k的页面大小已经出现了很多年了,就像文件系统上1k-4k的block_size一样,都是适应二十年前硬件的容量与速度而出现的,对于现在的硬件来说它们都显得太小了。使用更大的物理页,可以带来两个好处:TLB缓存命中率的提高和page_fault的次数降低。compaction正是为了支持THP而出现的。
在以前版本的内核中,要获得连续的物理内存只有一个办法:释放掉一部分内存,一般是释放page cache、脏页,或者进行页面swap。
而compaction提出了另外一个思路:重新组织内存。为此,提出了“可移动”页面的概念。在内核中的物理内存,有一部分是“可移动”的,内核使用的反碎片技术的基本原理,就是根据页的“可移动性”将页面分组。
那哪些页面是可以移动的呢? 非空闲的物理内存,当然要么是用户态进程在用,要么内核本身在用。对于前者,进程在访问物理内存的时候,实际上要通过页表的映射来访问。页表是一个可以做文章的地方:如果把一个页移动到另一个地方,如果可以同时修改页表,那么对应用程序就不会有影响。而对于内核访问物理内存时,是通过简单的常量偏移来做的。因此内核使用的物理页面无法移动。
定义了“可移动”的页面,具体到某一个页面,内核怎样知道它是否是可移动的?分配内存的函数,kmalloc,alloc_pages等在任何地方都可能被调用。内核又是怎样知道在这些地方分配的页面属于哪种类型呢?看这几个函数的原型
void *kmalloc(size_t size, gfp_t flags)
struct page * alloc_pages(gfp_t gfp_mask, unsigned int order)
内核自然不知道kmalloc分配的内存是作什么用途的,但是kernel 开发者知道,一个页面是否可移动,自然也是开发者们告诉内核的。gft_t中有个标志位:GFP_MOVABLE,开发者需要根据相应的内存是否要移动来设置该位。
了解了如何识别“可移动”页面,下面看看页面移动的流程:
1.???????? 锁定页,以避免在移动页的过程中有进程修改页面。页面记为oldpage
2.???????? 确保“writeback”已经完成
3.???????? 删除当前页面的全部映射,并将指向该页的页表项标记MIGRATION
4.???????? 查找新页,记为newpage
5.???????? 获取radix tree的锁,以阻塞所有试图通过radix tree来访问页面的进程。将radix tree中oldpage的指针指向newpage。释放radix tree的锁。
6.???????? 旧页的内容被拷到新页面中,设置新页面的各项标志
7.???????? 将所有页表项指向新页面
了解了compaction的目标和原理,那么该怎样查看系统中当前的碎片情况呢?/proc/pagetypeinfo文件提供了“可移动”和“不可移动”页面的分布数据, 一方面方便开发者调试,另一方面可以让系统管理员了解当前的系统运行状态。
Compaction在hadoop上所带来的性能问题,目前还不知道是在这种特定场景下才出现还是compaction本身就影响了性能。不过现在看来,在其它机器上还没有发现这种情况。
Compaction的目的是减少内存碎片,主要和THP搭配使用,适合需要大量连续内存的应用,比如KVM,能提升TLB效率和减少page fault次数,从而提高应用程序的执行效率。因此,去掉Compaction的支持,会对此类应用的性能所有影响。
参考:http://lwn.net/Articles/359158/你也许会喜欢:
- Mem Cgroup目录无法清理问题分析
- 深入剖析 linux GCC 4.4 的 STL string
- 利用 Flash 漏洞的木马程序分析报告 by 师兄
- 一个淘宝客劫持木马的分析
- 从Dump到POC系列一:Win32k内核提权漏洞分析
原文地址:hadoop集群System Cpu消耗过高问题分析 by 杂货店店长, 感谢原作者分享。

The steps to create and manage user accounts in MySQL are as follows: 1. Create a user: Use CREATEUSER'newuser'@'localhost'IDENTIFIEDBY'password'; 2. Assign permissions: Use GRANTSELECT, INSERT, UPDATEONmydatabase.TO'newuser'@'localhost'; 3. Fix permission error: Use REVOKEALLPRIVILEGESONmydatabase.FROM'newuser'@'localhost'; then reassign permissions; 4. Optimization permissions: Use SHOWGRA

MySQL is suitable for rapid development and small and medium-sized applications, while Oracle is suitable for large enterprises and high availability needs. 1) MySQL is open source and easy to use, suitable for web applications and small and medium-sized enterprises. 2) Oracle is powerful and suitable for large enterprises and government agencies. 3) MySQL supports a variety of storage engines, and Oracle provides rich enterprise-level functions.

The disadvantages of MySQL compared to other relational databases include: 1. Performance issues: You may encounter bottlenecks when processing large-scale data, and PostgreSQL performs better in complex queries and big data processing. 2. Scalability: The horizontal scaling ability is not as good as Google Spanner and Amazon Aurora. 3. Functional limitations: Not as good as PostgreSQL and Oracle in advanced functions, some functions require more custom code and maintenance.

MySQL supports four JOIN types: INNERJOIN, LEFTJOIN, RIGHTJOIN and FULLOUTERJOIN. 1.INNERJOIN is used to match rows in two tables and return results that meet the criteria. 2.LEFTJOIN returns all rows in the left table, even if the right table does not match. 3. RIGHTJOIN is opposite to LEFTJOIN and returns all rows in the right table. 4.FULLOUTERJOIN returns all rows in the two tables that meet or do not meet the conditions.

MySQL's performance under high load has its advantages and disadvantages compared with other RDBMSs. 1) MySQL performs well under high loads through the InnoDB engine and optimization strategies such as indexing, query cache and partition tables. 2) PostgreSQL provides efficient concurrent read and write through the MVCC mechanism, while Oracle and Microsoft SQLServer improve performance through their respective optimization strategies. With reasonable configuration and optimization, MySQL can perform well in high load environments.

InnoDBBufferPool reduces disk I/O by caching data and indexing pages, improving database performance. Its working principle includes: 1. Data reading: Read data from BufferPool; 2. Data writing: After modifying the data, write to BufferPool and refresh it to disk regularly; 3. Cache management: Use the LRU algorithm to manage cache pages; 4. Reading mechanism: Load adjacent data pages in advance. By sizing the BufferPool and using multiple instances, database performance can be optimized.

Compared with other programming languages, MySQL is mainly used to store and manage data, while other languages such as Python, Java, and C are used for logical processing and application development. MySQL is known for its high performance, scalability and cross-platform support, suitable for data management needs, while other languages have advantages in their respective fields such as data analytics, enterprise applications, and system programming.

MySQL is worth learning because it is a powerful open source database management system suitable for data storage, management and analysis. 1) MySQL is a relational database that uses SQL to operate data and is suitable for structured data management. 2) The SQL language is the key to interacting with MySQL and supports CRUD operations. 3) The working principle of MySQL includes client/server architecture, storage engine and query optimizer. 4) Basic usage includes creating databases and tables, and advanced usage involves joining tables using JOIN. 5) Common errors include syntax errors and permission issues, and debugging skills include checking syntax and using EXPLAIN commands. 6) Performance optimization involves the use of indexes, optimization of SQL statements and regular maintenance of databases.


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

VSCode Windows 64-bit Download
A free and powerful IDE editor launched by Microsoft

DVWA
Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

PhpStorm Mac version
The latest (2018.2.1) professional PHP integrated development tool

SublimeText3 English version
Recommended: Win version, supports code prompts!

Atom editor mac version download
The most popular open source editor