search
HomeDatabaseMysql TutorialMySQL多线程导入导出工具Mydumper

Mydumper是一个使用C语言编写的多线程导出导入工具,并且能够保证多个表之间的一致性.当然不是线程越多越好(这个跟服务器的配置等

今天从线上使用mysqldump将数据表从一个库导入到另外一个库,结果导出速度超级慢.于是网上搜了搜发现这个东东.测试之后发现还不错.分享给大家.

简单介绍一下
Mydumper是一个使用C语言编写的多线程导出导入工具,并且能够保证多个表之间的一致性.当然不是线程越多越好(这个跟服务器的配置等诸多因素有关,只能作为一个经验值而不是绝对值,机器好的时候,线程越多越好).

原理
前面提到保持数据一致性如何实现呢?
下面是官方给出的解答
主要是使用flush tables with read lock和start transaction with consistent snapshot,在flush tables with read lock时开启所有的线程,并且通过show master status和show slave status获得当前的position(便于使用Mydumper重建slave以及确保多个表之间的数据一致性)
原版如下:
This is all done following best MySQL practices and traditions:
1. Global write lock is acquired ("FLUSH TABLES WITH READ LOCK")
2. Various metadata is read ("SHOW SLAVE STATUS","SHOW MASTER STATUS")
3. Other threads connect and establish snapshots ("START TRANSACTION WITH CONSISTENT SNAPSHOT")
3.1. On pre-4.1.8 it creates dummy InnoDB table, and reads from it.
4. Once all worker threads announce the snapshot establishment, master executes "UNLOCK TABLES" and starts queueing jobs.
安装:
sudo yum install -y gcc gcc-c++ glib2-devel mysql-devel zlib-devel pcre-devel
cmake
make;make install
安装完会生产两个文件
[mysql@localhost ~]$ ls /usr/local/bin/
mydumper  myloader
例子:
导出
mysql> select count(*) from test;
+----------+
| count(*) |
+----------+
|  1048576 |
+----------+
1 row in set (0.41 sec)
[mysql@localhost bin]$ ./mydumper -u root -p 'xxxxxxxx' -t 4 -B test -T test -c --less-locking -o /home/mysql/
[mysql@localhost bin]$ ls /home/mysql/
test.test-schema.sql.gz  test.test.sql.gz
参数详解
-u "用户"
-p "密码"
-t "指定并行数,默认是4"
-B "指定DB"
-T "指定表"
-c "压缩"
--less-locking "尽量减少锁表锁定时间(针对InnoDB)"
-o "指定目录"

例如:
设置长查询的上限,如果存在比这个还长的查询则退出mydumper,也可以设置杀掉这个长查询
mydumper -u root -p 'xxxx' --long-query-guard 400 --kill-long-queries
通过regex设置正则表达,,需要设置db名字
mydumper -u root -p 'xxxx' --regex=test.name

导入
mysql> drop table test;
Query OK, 0 rows affected (0.26 sec)
mysql> exit
Bye
[mysql@localhost bin]$./myloader -u root -p 'xxxxx' -B test -d /home/mysql/
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| test          |
+----------------+
1 row in set (0.00 sec)

mysql> select count(*) from test;
+----------+
| count(*) |
+----------+
|  1048576 |
+----------+
1 row in set (0.39 sec)

结论:
Mydumper在导出导入过程因为可以多线程进行,因此速度上优于mysqldump.(题外话:导出上亿表数据的时候速度提升明显啊.^_^太爽了.大爱这个东东)
注意点:
--no-locks参数
这个参数官方给出英文注释"Do not execute the temporary shared read lock.  WARNING: This will cause inconsistent backups"也就是会导致备份不一致.
在导出myisam表时有表锁.所以先处理myisam表,记录myisam表个数,并在myisam表都处理完毕后,要立即解锁.尽量减少锁定的时间.

相关阅读:

Mydumper:MySQL多线程逻辑备份与恢复

MySQL备份工具mysqldump和mydumper的备份效率比较

linux

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
What are the different storage engines available in MySQL?What are the different storage engines available in MySQL?Apr 26, 2025 am 12:27 AM

MySQLoffersvariousstorageengines,eachsuitedfordifferentusecases:1)InnoDBisidealforapplicationsneedingACIDcomplianceandhighconcurrency,supportingtransactionsandforeignkeys.2)MyISAMisbestforread-heavyworkloads,lackingtransactionsupport.3)Memoryengineis

What are some common security vulnerabilities in MySQL?What are some common security vulnerabilities in MySQL?Apr 26, 2025 am 12:27 AM

Common security vulnerabilities in MySQL include SQL injection, weak passwords, improper permission configuration, and unupdated software. 1. SQL injection can be prevented by using preprocessing statements. 2. Weak passwords can be avoided by forcibly using strong password strategies. 3. Improper permission configuration can be resolved through regular review and adjustment of user permissions. 4. Unupdated software can be patched by regularly checking and updating the MySQL version.

How can you identify slow queries in MySQL?How can you identify slow queries in MySQL?Apr 26, 2025 am 12:15 AM

Identifying slow queries in MySQL can be achieved by enabling slow query logs and setting thresholds. 1. Enable slow query logs and set thresholds. 2. View and analyze slow query log files, and use tools such as mysqldumpslow or pt-query-digest for in-depth analysis. 3. Optimizing slow queries can be achieved through index optimization, query rewriting and avoiding the use of SELECT*.

How can you monitor MySQL server health and performance?How can you monitor MySQL server health and performance?Apr 26, 2025 am 12:15 AM

To monitor the health and performance of MySQL servers, you should pay attention to system health, performance metrics and query execution. 1) Monitor system health: Use top, htop or SHOWGLOBALSTATUS commands to view CPU, memory, disk I/O and network activities. 2) Track performance indicators: monitor key indicators such as query number per second, average query time and cache hit rate. 3) Ensure query execution optimization: Enable slow query logs, record and optimize queries whose execution time exceeds the set threshold.

Compare and contrast MySQL and MariaDB.Compare and contrast MySQL and MariaDB.Apr 26, 2025 am 12:08 AM

The main difference between MySQL and MariaDB is performance, functionality and license: 1. MySQL is developed by Oracle, and MariaDB is its fork. 2. MariaDB may perform better in high load environments. 3.MariaDB provides more storage engines and functions. 4.MySQL adopts a dual license, and MariaDB is completely open source. The existing infrastructure, performance requirements, functional requirements and license costs should be taken into account when choosing.

How does MySQL's licensing compare to other database systems?How does MySQL's licensing compare to other database systems?Apr 25, 2025 am 12:26 AM

MySQL uses a GPL license. 1) The GPL license allows the free use, modification and distribution of MySQL, but the modified distribution must comply with GPL. 2) Commercial licenses can avoid public modifications and are suitable for commercial applications that require confidentiality.

When would you choose InnoDB over MyISAM, and vice versa?When would you choose InnoDB over MyISAM, and vice versa?Apr 25, 2025 am 12:22 AM

The situations when choosing InnoDB instead of MyISAM include: 1) transaction support, 2) high concurrency environment, 3) high data consistency; conversely, the situation when choosing MyISAM includes: 1) mainly read operations, 2) no transaction support is required. InnoDB is suitable for applications that require high data consistency and transaction processing, such as e-commerce platforms, while MyISAM is suitable for read-intensive and transaction-free applications such as blog systems.

Explain the purpose of foreign keys in MySQL.Explain the purpose of foreign keys in MySQL.Apr 25, 2025 am 12:17 AM

In MySQL, the function of foreign keys is to establish the relationship between tables and ensure the consistency and integrity of the data. Foreign keys maintain the effectiveness of data through reference integrity checks and cascading operations. Pay attention to performance optimization and avoid common errors when using them.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

SAP NetWeaver Server Adapter for Eclipse

SAP NetWeaver Server Adapter for Eclipse

Integrate Eclipse with SAP NetWeaver application server.

Atom editor mac version download

Atom editor mac version download

The most popular open source editor

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function

SublimeText3 English version

SublimeText3 English version

Recommended: Win version, supports code prompts!

MantisBT

MantisBT

Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.