HBase解决Region Server Compact过程占用大量网络出口带宽的问题-Mysql Tutorial-php.cn

Home

Database

Mysql Tutorial

HBase解决Region Server Compact过程占用大量网络出口带宽的问题

WBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWB

Jun 07, 2016 pm 05:56 PM

compacthbaseserversolve

HBase解决Region Server Compact过程占用大量网络出口带宽的问题作者: 大圆那些事 | 文章可以转载，请以超链接形式标明文章原始出处和作者信息网址: 推荐：《HBase权威指南》 HBase 0.92版本之后，Region Server的Compact过程根据待合并的文件大小分为smal

HBase解决Region Server Compact过程占用大量网络出口带宽的问题

作者: 大圆那些事 | 文章可以转载，请以超链接形式标明文章原始出处和作者信息

网址:

推荐：《HBase权威指南》

HBase 0.92版本之后，Region Server的Compact过程根据待合并的文件大小分为small compaction和large compaction两种，由此可能导致在集群写入量大的时候Compact占用过多的网络出口带宽。本文将详细描述集群使用过程中遇到这一问题的排查过程及其解决方法。

1. 发现问题

HBase集群（版本为0.94.0）运行过程中，发现5台Region Server的网络出口带宽经常维持在100MB/s以上，香港虚拟主机，接近到网卡的极限；同时Region Server的机器load负载也很高，高峰时候能够达到30~50。

2. 排查问题

1、集群实际运行过程中，观察到Region Server服务端的网卡，平均每台写入流量大概60MB/s（此时写入量已经很大了）；读出流量90MB/s，有时甚至突破100MB/s（注：每台机器都是千兆网卡）；

2、观察实际的写入数据量在每秒5w tps左右，单条记录平均大小为1KB，大概会占用50MB/s左右的网卡入口带宽请求量，和观察到的现象一致；

3、观察查询量在每秒6w qps左右，单条记录平均大小为1KB，大概会占用60MB/s左右的网卡出口带宽请求量，奇怪的是实际观察到有接近甚至超过100MB/s的网络出口带宽请求量，多出了40MB/s左右的网络出口带宽；

4、经分析排查确定导致上述过程的原因，可能是HBase服务端由于写入量过大频繁触发compaction过程，而compaction是需要读HBase数据的，因此占据了相当部分的网络出口带宽；

5、结合对相关源码org/apache/hadoop/hbase/regionserver/CompactSplitThread.java的分析，决定对HBase集群配置做出变更（具体见下一小节），主要目的是减少compaction的发生；

6、接下来，观察到Region Server的网络利用率明显降低，一般进出口带宽能维持在70MB/s以下。

3. 解决问题

HBase 0.92版本之后增加了关于compact的配置选项，compact分为small compaction和large compaction两个线程池执行（默认都是各有1个线程，虚拟主机，具体源代码见：org/apache/hadoop/hbase/regionserver/CompactSplitThread.java），由于compact过程需要从HBase集群读取数据，因此实际运行中导致了compact占用大量网络出口流量，解决方案为选择性地关闭small compaction或large compaction。有以下两种变更方式均可解决：

1）方案一

（1）修改hbase.regionserver.thread.compaction.throttle为一个很大的值（如50GB），香港虚拟主机，强制让所有compact都变为small compaction，减少compact的压力；

（2）将small compaction和large compaction线程数均设置为1，减少compact的压力（可不配置，系统默认也会将其初始化为1）。

操作步骤：

准备hbase-site.xml文件，添加或修改如下选项：

hbase.regionserver.thread.compaction.throttle53687091200hbase.regionserver.thread.compaction.small1hbase.regionserver.thread.compaction.large1

重启集群使配置生效。

2）方案二

将small compaction线程数均设置为0，从而关闭small compaction，只剩下large compaction，也可减少compact的压力。

操作步骤：

准备hbase-site.xml文件，添加或修改如下选项：

hbase.regionserver.thread.compaction.small0

重启集群使配置生效。

posted on

Statement

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Explain the InnoDB Buffer Pool and its importance for performance.Apr 19, 2025 am 12:24 AM

InnoDBBufferPool reduces disk I/O by caching data and indexing pages, improving database performance. Its working principle includes: 1. Data reading: Read data from BufferPool; 2. Data writing: After modifying the data, write to BufferPool and refresh it to disk regularly; 3. Cache management: Use the LRU algorithm to manage cache pages; 4. Reading mechanism: Load adjacent data pages in advance. By sizing the BufferPool and using multiple instances, database performance can be optimized.

MySQL vs. Other Programming Languages: A ComparisonApr 19, 2025 am 12:22 AM

Compared with other programming languages, MySQL is mainly used to store and manage data, while other languages such as Python, Java, and C are used for logical processing and application development. MySQL is known for its high performance, scalability and cross-platform support, suitable for data management needs, while other languages have advantages in their respective fields such as data analytics, enterprise applications, and system programming.

Learning MySQL: A Step-by-Step Guide for New UsersApr 19, 2025 am 12:19 AM

MySQL is worth learning because it is a powerful open source database management system suitable for data storage, management and analysis. 1) MySQL is a relational database that uses SQL to operate data and is suitable for structured data management. 2) The SQL language is the key to interacting with MySQL and supports CRUD operations. 3) The working principle of MySQL includes client/server architecture, storage engine and query optimizer. 4) Basic usage includes creating databases and tables, and advanced usage involves joining tables using JOIN. 5) Common errors include syntax errors and permission issues, and debugging skills include checking syntax and using EXPLAIN commands. 6) Performance optimization involves the use of indexes, optimization of SQL statements and regular maintenance of databases.

MySQL: Essential Skills for Beginners to MasterApr 18, 2025 am 12:24 AM

MySQL is suitable for beginners to learn database skills. 1. Install MySQL server and client tools. 2. Understand basic SQL queries, such as SELECT. 3. Master data operations: create tables, insert, update, and delete data. 4. Learn advanced skills: subquery and window functions. 5. Debugging and optimization: Check syntax, use indexes, avoid SELECT*, and use LIMIT.

MySQL: Structured Data and Relational DatabasesApr 18, 2025 am 12:22 AM

MySQL efficiently manages structured data through table structure and SQL query, and implements inter-table relationships through foreign keys. 1. Define the data format and type when creating a table. 2. Use foreign keys to establish relationships between tables. 3. Improve performance through indexing and query optimization. 4. Regularly backup and monitor databases to ensure data security and performance optimization.

MySQL: Key Features and Capabilities ExplainedApr 18, 2025 am 12:17 AM

MySQL is an open source relational database management system that is widely used in Web development. Its key features include: 1. Supports multiple storage engines, such as InnoDB and MyISAM, suitable for different scenarios; 2. Provides master-slave replication functions to facilitate load balancing and data backup; 3. Improve query efficiency through query optimization and index use.

The Purpose of SQL: Interacting with MySQL DatabasesApr 18, 2025 am 12:12 AM

SQL is used to interact with MySQL database to realize data addition, deletion, modification, inspection and database design. 1) SQL performs data operations through SELECT, INSERT, UPDATE, DELETE statements; 2) Use CREATE, ALTER, DROP statements for database design and management; 3) Complex queries and data analysis are implemented through SQL to improve business decision-making efficiency.

MySQL for Beginners: Getting Started with Database ManagementApr 18, 2025 am 12:10 AM

The basic operations of MySQL include creating databases, tables, and using SQL to perform CRUD operations on data. 1. Create a database: CREATEDATABASEmy_first_db; 2. Create a table: CREATETABLEbooks(idINTAUTO_INCREMENTPRIMARYKEY, titleVARCHAR(100)NOTNULL, authorVARCHAR(100)NOTNULL, published_yearINT); 3. Insert data: INSERTINTObooks(title, author, published_year)VA

See all articles