search
HomeDatabaseMysql TutorialMySQL多线程导入导出工具Mydumper

Mydumper是一个使用C语言编写的多线程导出导入工具,并且能够保证多个表之间的一致性.当然不是线程越多越好(这个跟服务器的配置等

今天从线上使用mysqldump将数据表从一个库导入到另外一个库,结果导出速度超级慢.于是网上搜了搜发现这个东东.测试之后发现还不错.分享给大家.

简单介绍一下
Mydumper是一个使用C语言编写的多线程导出导入工具,并且能够保证多个表之间的一致性.当然不是线程越多越好(这个跟服务器的配置等诸多因素有关,只能作为一个经验值而不是绝对值,机器好的时候,线程越多越好).

原理
前面提到保持数据一致性如何实现呢?
下面是官方给出的解答
主要是使用flush tables with read lock和start transaction with consistent snapshot,在flush tables with read lock时开启所有的线程,并且通过show master status和show slave status获得当前的position(便于使用Mydumper重建slave以及确保多个表之间的数据一致性)
原版如下:
This is all done following best MySQL practices and traditions:
1. Global write lock is acquired ("FLUSH TABLES WITH READ LOCK")
2. Various metadata is read ("SHOW SLAVE STATUS","SHOW MASTER STATUS")
3. Other threads connect and establish snapshots ("START TRANSACTION WITH CONSISTENT SNAPSHOT")
3.1. On pre-4.1.8 it creates dummy InnoDB table, and reads from it.
4. Once all worker threads announce the snapshot establishment, master executes "UNLOCK TABLES" and starts queueing jobs.
安装:
sudo yum install -y gcc gcc-c++ glib2-devel mysql-devel zlib-devel pcre-devel
cmake
make;make install
安装完会生产两个文件
[mysql@localhost ~]$ ls /usr/local/bin/
mydumper  myloader
例子:
导出
mysql> select count(*) from test;
+----------+
| count(*) |
+----------+
|  1048576 |
+----------+
1 row in set (0.41 sec)
[mysql@localhost bin]$ ./mydumper -u root -p 'xxxxxxxx' -t 4 -B test -T test -c --less-locking -o /home/mysql/
[mysql@localhost bin]$ ls /home/mysql/
test.test-schema.sql.gz  test.test.sql.gz
参数详解
-u "用户"
-p "密码"
-t "指定并行数,默认是4"
-B "指定DB"
-T "指定表"
-c "压缩"
--less-locking "尽量减少锁表锁定时间(针对InnoDB)"
-o "指定目录"

例如:
设置长查询的上限,如果存在比这个还长的查询则退出mydumper,也可以设置杀掉这个长查询
mydumper -u root -p 'xxxx' --long-query-guard 400 --kill-long-queries
通过regex设置正则表达,,需要设置db名字
mydumper -u root -p 'xxxx' --regex=test.name

导入
mysql> drop table test;
Query OK, 0 rows affected (0.26 sec)
mysql> exit
Bye
[mysql@localhost bin]$./myloader -u root -p 'xxxxx' -B test -d /home/mysql/
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| test          |
+----------------+
1 row in set (0.00 sec)

mysql> select count(*) from test;
+----------+
| count(*) |
+----------+
|  1048576 |
+----------+
1 row in set (0.39 sec)

结论:
Mydumper在导出导入过程因为可以多线程进行,因此速度上优于mysqldump.(题外话:导出上亿表数据的时候速度提升明显啊.^_^太爽了.大爱这个东东)
注意点:
--no-locks参数
这个参数官方给出英文注释"Do not execute the temporary shared read lock.  WARNING: This will cause inconsistent backups"也就是会导致备份不一致.
在导出myisam表时有表锁.所以先处理myisam表,记录myisam表个数,并在myisam表都处理完毕后,要立即解锁.尽量减少锁定的时间.

相关阅读:

Mydumper:MySQL多线程逻辑备份与恢复

MySQL备份工具mysqldump和mydumper的备份效率比较

linux

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Reduce the use of MySQL memory in DockerReduce the use of MySQL memory in DockerMar 04, 2025 pm 03:52 PM

This article explores optimizing MySQL memory usage in Docker. It discusses monitoring techniques (Docker stats, Performance Schema, external tools) and configuration strategies. These include Docker memory limits, swapping, and cgroups, alongside

How to solve the problem of mysql cannot open shared libraryHow to solve the problem of mysql cannot open shared libraryMar 04, 2025 pm 04:01 PM

This article addresses MySQL's "unable to open shared library" error. The issue stems from MySQL's inability to locate necessary shared libraries (.so/.dll files). Solutions involve verifying library installation via the system's package m

How do you alter a table in MySQL using the ALTER TABLE statement?How do you alter a table in MySQL using the ALTER TABLE statement?Mar 19, 2025 pm 03:51 PM

The article discusses using MySQL's ALTER TABLE statement to modify tables, including adding/dropping columns, renaming tables/columns, and changing column data types.

Run MySQl in Linux (with/without podman container with phpmyadmin)Run MySQl in Linux (with/without podman container with phpmyadmin)Mar 04, 2025 pm 03:54 PM

This article compares installing MySQL on Linux directly versus using Podman containers, with/without phpMyAdmin. It details installation steps for each method, emphasizing Podman's advantages in isolation, portability, and reproducibility, but also

What is SQLite? Comprehensive overviewWhat is SQLite? Comprehensive overviewMar 04, 2025 pm 03:55 PM

This article provides a comprehensive overview of SQLite, a self-contained, serverless relational database. It details SQLite's advantages (simplicity, portability, ease of use) and disadvantages (concurrency limitations, scalability challenges). C

How do I configure SSL/TLS encryption for MySQL connections?How do I configure SSL/TLS encryption for MySQL connections?Mar 18, 2025 pm 12:01 PM

Article discusses configuring SSL/TLS encryption for MySQL, including certificate generation and verification. Main issue is using self-signed certificates' security implications.[Character count: 159]

Running multiple MySQL versions on MacOS: A step-by-step guideRunning multiple MySQL versions on MacOS: A step-by-step guideMar 04, 2025 pm 03:49 PM

This guide demonstrates installing and managing multiple MySQL versions on macOS using Homebrew. It emphasizes using Homebrew to isolate installations, preventing conflicts. The article details installation, starting/stopping services, and best pra

What are some popular MySQL GUI tools (e.g., MySQL Workbench, phpMyAdmin)?What are some popular MySQL GUI tools (e.g., MySQL Workbench, phpMyAdmin)?Mar 21, 2025 pm 06:28 PM

Article discusses popular MySQL GUI tools like MySQL Workbench and phpMyAdmin, comparing their features and suitability for beginners and advanced users.[159 characters]

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
1 months agoBy尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

SAP NetWeaver Server Adapter for Eclipse

SAP NetWeaver Server Adapter for Eclipse

Integrate Eclipse with SAP NetWeaver application server.

MinGW - Minimalist GNU for Windows

MinGW - Minimalist GNU for Windows

This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

VSCode Windows 64-bit Download

VSCode Windows 64-bit Download

A free and powerful IDE editor launched by Microsoft

MantisBT

MantisBT

Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),