Detailed explanation of mysql master-slave synchronization problem and solution process

A mysql master-slave synchronization solution process

The table structure was modified the day before yesterday, and one of the tables was extended with a field structure, starting from varchar(30) Expanded to varchar(50), the table data is more than 1.2 million. It only takes 40 seconds to execute in the main database, but it takes 4 hours to synchronize from the slave database.

Although the main library executes very quickly, the number of rows affected is 1.2 million rows. The slave library synchronizes the structural changes of 1.2 million rows, instead of simply executing sql commands to modify the slave library.
I didn’t notice it at first, but later when the business was slow, I started to feel something was wrong, so I quickly went to mysql to check the currently blocked mysql process:

show proccesslist

Here The result is not the result at that time (many queries were blocked at that time):

| Id     | User  | Host            | db   | Command     | Time   | State                                                                 | Info             |
+--------+-------+-----------------+------+-------------+--------+-----------------------------------------------------------------------+------------------+
| 722874 | bakup | 127.0.0.1:36759 | NULL | Binlog Dump | 281055 | Master has sent all binlog to slave; waiting for binlog to be updated | NULL             |
| 991867 | root  | localhost       | NULL | Sleep       |    780 |                                                                       | NULL             |
| 992585 | root  | localhost       | NULL | Query       |      0 | NULL                                                                  | show processlist |

1.Id:Process id, very useful when you want to kill a statement.

2.User: Display the previous user. If you are not root, this command will only display the sql statements within your authority.

3.Host: Display which IP and port this statement is issued from

4.db: Display which process this process is currently connected to Database

5.Command:Displays the executed commands of the current connection, sleep, query, connect, binlog (master-slave)

6.Time: The duration of this state, the unit is seconds.

7.State:Displays the status of the sql statement using the current connection. It is a very important column. There will be descriptions of all states later. Please note that state is only a certain state in the execution of the statement. A sql statement, for example, has been queried. It may need to go through copying to tmp table, Sorting result, Sending data and other states before it can be completed.

8.info:Display this sql statement

Now we are killing the blocking process, that is, the process that synchronously modifies the structure

kill 722874

We were able to resume normal business queries, but a new problem came. The master and slave were forcibly suspended, and an error occurred. The master database could not be synchronized to the slave database, and the latest business query data could not be synchronized. .

Go up to the query command from the database (the result here is not the result at that time (it was an error message at that time)):

(Mon Jun 26 20:49:40 2017) db_2 >>show slave status\G*************************** 1. row ***************************   Slave_IO_State: Waiting for master to send event  Master_Host: 127.0.0.1  Master_User: bakup
                  Master_Port: 3306Connect_Retry: 60  Master_Log_File: mysql-bin.000330  Read_Master_Log_Pos: 445043216   Relay_Log_File: 174-relay-bin.000043Relay_Log_Pos: 445043362Relay_Master_Log_File: mysql-bin.000330 Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: information_schema,mysql,performance_schema,test,zabbix,information_schema,mysql,performance_schema,test,zabbix
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0   Last_Error: 
                 Skip_Counter: 0  Exec_Master_Log_Pos: 445043216  Relay_Log_Space: 445043559  Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0   Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0Last_IO_Error: 
               Last_SQL_Errno: 0   Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 11 row in set (0.00 sec)

So we consulted the operation and maintenance and took the following action The following method:

 恢复主库到改变字段前的状态
2 停止主从二进制日志的写入，主从同步停止
3 开始改变主库字段结构
4 改变从库字段结构（注意此时主从同步已经停止）
5 修正此前发生的同步错误
6 恢复主从二进制日志的写入
7 重新开启主从同步

The problem is solved in about 40 minutes.

This operation is also a bit urgent. It should be better to make structural changes to large amounts of data at night when the background is hardly accessed. An assessment was also conducted on the same day, and it could be successful within 2 hours.

Attached, state column information:

Checking table
　正在检查数据表（这是自动的）。
Closing tables
　正在将表中修改的数据刷新到磁盘中，同时正在关闭已经用完的表。这是一个很快的操作，如果不是这样的话，就应该确认磁盘空间是否已经满了或者磁盘是否正处于重负中。
Connect Out
　复制从服务器正在连接主服务器。
Copying to tmp table on disk
　由于临时结果集大于tmp_table_size，正在将临时表从内存存储转为磁盘存储以此节省内存。
Creating tmp table
　正在创建临时表以存放部分查询结果。
deleting from main table
　服务器正在执行多表删除中的第一部分，刚删除第一个表。
deleting from reference tables
　服务器正在执行多表删除中的第二部分，正在删除其他表的记录。
Flushing tables
　正在执行FLUSH TABLES，等待其他线程关闭数据表。
Killed
　发送了一个kill请求给某线程，那么这个线程将会检查kill标志位，同时会放弃下一个kill请求。MySQL会在每次的主循环中检查kill标志位，不过有些情况下该线程可能会过一小段才能死掉。如果该线程程被其他线程锁住了，那么kill请求会在锁释放时马上生效。
Locked
　被其他查询锁住了。
Sending data
　正在处理SELECT查询的记录，同时正在把结果发送给客户端。
Sorting for group
　正在为GROUP BY做排序。
　Sorting for order
　正在为ORDER BY做排序。
Opening tables
　这个过程应该会很快，除非受到其他因素的干扰。例如，在执ALTER TABLE或LOCK TABLE语句行完以前，数据表无法被其他线程打开。正尝试打开一个表。
Removing duplicates
　正在执行一个SELECT DISTINCT方式的查询，但是MySQL无法在前一个阶段优化掉那些重复的记录。因此，MySQL需要再次去掉重复的记录，然后再把结果发送给客户端。
Reopen table
　获得了对一个表的锁，但是必须在表结构修改之后才能获得这个锁。已经释放锁，关闭数据表，正尝试重新打开数据表。
Repair by sorting
　修复指令正在排序以创建索引。
Repair with keycache
　修复指令正在利用索引缓存一个一个地创建新索引。它会比Repair by sorting慢些。
Searching rows for update
　正在讲符合条件的记录找出来以备更新。它必须在UPDATE要修改相关的记录之前就完成了。
Sleeping
　正在等待客户端发送新请求.
System lock　正在等待取得一个外部的系统锁。如果当前没有运行多个mysqld服务器同时请求同一个表，那么可以通过增加--skip-external-locking参数来禁止外部系统锁。
Upgrading lock　INSERT DELAYED正在尝试取得一个锁表以插入新记录。
Updating
　正在搜索匹配的记录，并且修改它们。
User Lock
　正在等待GET_LOCK()。
Waiting for tables
　该线程得到通知，数据表结构已经被修改了，需要重新打开数据表以取得新的结构。然后，为了能的重新打开数据表，必须等到所有其他线程关闭这个表。以下几种情况下会产生这个通知：FLUSH TABLES tbl_name, ALTER TABLE, RENAME TABLE, REPAIR TABLE, ANALYZE TABLE,或OPTIMIZE TABLE。
waiting for handler insert
　INSERT DELAYED已经处理完了所有待处理的插入操作，正在等待新的请求。

The above is the detailed content of Detailed explanation of mysql master-slave synchronization problem and solution process. For more information, please follow other related articles on the PHP Chinese website!