五一前，一个DBA同事反馈，在日常环境中删除一个大的slow log文件（假设文件大小10G以上吧），然后在MySQL中执行flush slow logs，会发现mysqld hang住。
今天尝试着重现了此问题，这里简要分析下原因。
重现步骤：
1. 构造slow log （将long_query_time设成了0）；
2. 观察删rm slow log瞬间，tps/qps变化；
3. 观察执行flush slow logs瞬间，tps/qps变化；
4. 记录flush slow logs执行时， pstack打出的调用栈情况；
第一步，没啥好说的。
第二步，tps/qps没啥变化。
第三步，会发现tps/qps瞬间跌0，如下所示：
[ 639s] threads: 32, tps: 1121.00, reads/s: 15843.98, writes/s: 4490.99[ 640s] threads: 32, tps: 792.99, reads/s: 10803.89, writes/s: 3150.97[ 641s] threads: 32, tps: 0.00, reads/s: 0.00, writes/s: 0.00[ 642s] threads: 32, tps: 0.00, reads/s: 0.00, writes/s: 0.00[ 643s] threads: 32, tps: 471.01, reads/s: 6860.08, writes/s: 1908.02
mysql命令行会发现，flush slow logs执行时间刚好为3s左右。
第四步，我们看下pstack的输出结果，只记录相关的：
610 Thread 5 (Thread 0x2afdc4101700 (LWP 30762)):611 #00x0000003c6e40a7d6 in pthread_rwlock_rdlock () from /lib64/libpthread.so.0612 #10x0000000000825135 in inline_mysql_rwlock_rdlock ()613 #20x0000000000838004 in LOGGER::lock_shared() ()614 #30x00000000008283bf in LOGGER::slow_log_print(THD*, char const*, unsigned int, unsigned long long) ()615 #40x0000000000832b30 in slow_log_print(THD*, char const*, unsigned int, unsigned long long) ()616 #50x0000000000609f23 in log_slow_statement(THD*) ()617 #60x00000000006099d1 in dispatch_command(enum_server_command, THD*, char*, unsigned int) ()618 #70x0000000000606e02 in do_command(THD*) ()619 #80x00000000006f070f in do_handle_one_connection(THD*) ()620 #90x00000000006f020d in handle_one_connection ()621 #10 0x0000003c6e4077f1 in start_thread () from /lib64/libpthread.so.0622 #11 0x0000003c6e0e570d in clone () from /lib64/libc.so.6623 Thread 4 (Thread 0x2afdd0080700 (LWP 30763)):624 #00x0000003c6e40a7d6 in pthread_rwlock_rdlock () from /lib64/libpthread.so.0625 #10x0000000000825135 in inline_mysql_rwlock_rdlock ()626 #20x0000000000838004 in LOGGER::lock_shared() ()627 #30x00000000008283bf in LOGGER::slow_log_print(THD*, char const*, unsigned int, unsigned long long) ()628 #40x0000000000832b30 in slow_log_print(THD*, char const*, unsigned int, unsigned long long) ()629 #50x0000000000609f23 in log_slow_statement(THD*) ()630 #60x00000000006099d1 in dispatch_command(enum_server_command, THD*, char*, unsigned int) ()631 #70x0000000000606e02 in do_command(THD*) ()632 #80x00000000006f070f in do_handle_one_connection(THD*) ()633 #90x00000000006f020d in handle_one_connection ()634 #10 0x0000003c6e4077f1 in start_thread () from /lib64/libpthread.so.0635 #11 0x0000003c6e0e570d in clone () from /lib64/libc.so.6636 Thread 3 (Thread 0x2afdd0101700 (LWP 30764)):637 #00x0000003c6e40a7d6 in pthread_rwlock_rdlock () from /lib64/libpthread.so.0638 #10x0000000000825135 in inline_mysql_rwlock_rdlock ()639 #20x0000000000838004 in LOGGER::lock_shared() ()640 #30x00000000008283bf in LOGGER::slow_log_print(THD*, char const*, unsigned int, unsigned long long) ()641 #40x0000000000832b30 in slow_log_print(THD*, char const*, unsigned int, unsigned long long) ()642 #50x0000000000609f23 in log_slow_statement(THD*) ()643 #60x00000000006099d1 in dispatch_command(enum_server_command, THD*, char*, unsigned int) ()644 #70x0000000000606e02 in do_command(THD*) ()645 #80x00000000006f070f in do_handle_one_connection(THD*) ()646 #90x00000000006f020d in handle_one_connection ()647 #10 0x0000003c6e4077f1 in start_thread () from /lib64/libpthread.so.0648 #11 0x0000003c6e0e570d in clone () from /lib64/libc.so.6649 Thread 2 (Thread 0x2afe18080700 (LWP 30855)):650 #00x0000003c6e40e54d in close () from /lib64/libpthread.so.0651 #10x00000000008f56ed in my_close ()652 #20x0000000000825c16 in inline_mysql_file_close ()653 #30x000000000082b305 in MYSQL_LOG::close(unsigned int) ()654 #40x000000000082b634 in MYSQL_QUERY_LOG::reopen_file() ()655 #50x0000000000828283 in LOGGER::flush_slow_log() ()656 #60x000000000071d8fc in reload_acl_and_cache(THD*, unsigned long, TABLE_LIST*, int*) ()657 #70x0000000000610200 in mysql_execute_command(THD*) ()658 #80x000000000061534d in mysql_parse(THD*, char*, unsigned int, Parser_state*) ()659 #90x00000000006086a0 in dispatch_command(enum_server_command, THD*, char*, unsigned int) ()660 #10 0x0000000000606e02 in do_command(THD*) ()661 #11 0x00000000006f070f in do_handle_one_connection(THD*) ()662 #12 0x00000000006f020d in handle_one_connection ()663 #13 0x0000003c6e4077f1 in start_thread () from /lib64/libpthread.so.0664 #14 0x0000003c6e0e570d in clone () from /lib64/libc.so.6
会发现Thread 2在执行flush slow logs操作，其他的线程都在等待锁LOCK_log上边。
背后的原因其实很简单，在shell中执行rm slow log操作时，由于mysqld仍有文件句柄打开此文件，所以实际上此时文件并未删除。执行flush logs操作，其实际执行的是1)close；2）open 操作（logger.flush_slow_log -> mysql_slow_log.reopen_file），在close操作执行时，文件系统真正删除文件，此时该线程占用着LOCK_log锁。
删除时会执行刷脏（当然我构造这个场景很极端，基本所有slow log文件的内容都在文件系统缓存中），这个会很耗时间，比如我执行这个语句耗了3s。此时间段内，如果连接发来的语句需要记log（server层的log:slow log/binlog/general log共有LOCK_log这把锁）就会处于等待状态，那么系统对外的反应就是hang住了。
flush slow logs中调用执行的close所需时间和文件大小、以及文件系统缓存中该文件脏页比例都有关系，比如我在执行flush slow logs前使用sysctl vm.drop_caches=3清空
了文件系统缓存的话，同样大小的flush slow logs操作执行时间是0.42s，相应的阻塞时间也会减少不少。
可以考虑在slow logs的文件句柄上执行posix_fadvise调用，促使不会缓存很大的log文件内容（slow log也没啥需要缓存的），这有篇霸爷的文章，可以参考下 posix_fadvise清除缓存的误解和改进措施 。
另外，peter在07年就讨论过这个问题， Be careful rotating MySQL logs 其给出的建议是先mv file，然后flush logs，再执行删除文件的操作，让真正的删除行为由自己而不是mysqld完成。比较遗憾的是，五年过去了，LOCK_log这把锁的问题还没有完整的解决掉。
PS：
文章结尾记一点备忘，通过close/rm操作删除一个10G大小的文件，在执行sysctl vm.drop_caches=3清空缓存后，此操作的耗时仍在百毫秒量级（我的机器上是200ms+），其背后做了什么事情还需要找内核组的同事了解下。
MySQL's Role: Databases in Web ApplicationsApr 17, 2025 am 12:23 AMThe main role of MySQL in web applications is to store and manage data. 1.MySQL efficiently processes user information, product catalogs, transaction records and other data. 2. Through SQL query, developers can extract information from the database to generate dynamic content. 3.MySQL works based on the client-server model to ensure acceptable query speed.
MySQL: Building Your First DatabaseApr 17, 2025 am 12:22 AMThe steps to build a MySQL database include: 1. Create a database and table, 2. Insert data, and 3. Conduct queries. First, use the CREATEDATABASE and CREATETABLE statements to create the database and table, then use the INSERTINTO statement to insert the data, and finally use the SELECT statement to query the data.
MySQL: A Beginner-Friendly Approach to Data StorageApr 17, 2025 am 12:21 AMMySQL is suitable for beginners because it is easy to use and powerful. 1.MySQL is a relational database, and uses SQL for CRUD operations. 2. It is simple to install and requires the root user password to be configured. 3. Use INSERT, UPDATE, DELETE, and SELECT to perform data operations. 4. ORDERBY, WHERE and JOIN can be used for complex queries. 5. Debugging requires checking the syntax and use EXPLAIN to analyze the query. 6. Optimization suggestions include using indexes, choosing the right data type and good programming habits.
Is MySQL Beginner-Friendly? Assessing the Learning CurveApr 17, 2025 am 12:19 AMMySQL is suitable for beginners because: 1) easy to install and configure, 2) rich learning resources, 3) intuitive SQL syntax, 4) powerful tool support. Nevertheless, beginners need to overcome challenges such as database design, query optimization, security management, and data backup.
Is SQL a Programming Language? Clarifying the TerminologyApr 17, 2025 am 12:17 AMYes,SQLisaprogramminglanguagespecializedfordatamanagement.1)It'sdeclarative,focusingonwhattoachieveratherthanhow.2)SQLisessentialforquerying,inserting,updating,anddeletingdatainrelationaldatabases.3)Whileuser-friendly,itrequiresoptimizationtoavoidper
Explain the ACID properties (Atomicity, Consistency, Isolation, Durability).Apr 16, 2025 am 12:20 AMACID attributes include atomicity, consistency, isolation and durability, and are the cornerstone of database design. 1. Atomicity ensures that the transaction is either completely successful or completely failed. 2. Consistency ensures that the database remains consistent before and after a transaction. 3. Isolation ensures that transactions do not interfere with each other. 4. Persistence ensures that data is permanently saved after transaction submission.
MySQL: Database Management System vs. Programming LanguageApr 16, 2025 am 12:19 AMMySQL is not only a database management system (DBMS) but also closely related to programming languages. 1) As a DBMS, MySQL is used to store, organize and retrieve data, and optimizing indexes can improve query performance. 2) Combining SQL with programming languages, embedded in Python, using ORM tools such as SQLAlchemy can simplify operations. 3) Performance optimization includes indexing, querying, caching, library and table division and transaction management.
MySQL: Managing Data with SQL CommandsApr 16, 2025 am 12:19 AMMySQL uses SQL commands to manage data. 1. Basic commands include SELECT, INSERT, UPDATE and DELETE. 2. Advanced usage involves JOIN, subquery and aggregate functions. 3. Common errors include syntax, logic and performance issues. 4. Optimization tips include using indexes, avoiding SELECT* and using LIMIT.
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
SublimeText3 English version
Recommended: Win version, supports code prompts!
mPDF
mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),
MinGW - Minimalist GNU for Windows
This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.
SublimeText3 Chinese version
Chinese version, very easy to use
SublimeText3 Mac version
God-level code editing software (SublimeText3)
