In-depth understanding of the master-slave synchronization mechanism in Redis

This article will take you to understand the master-slave synchronization in Redis, and introduce the two structural models of Redis master-slave, the establishment of the master-slave relationship, the master-slave replication strategy, etc. I hope it will be useful to everyone. Helped!

#The previous article analyzed the characteristics and core principles of redis in detail. From this article, we will analyze and interpret the deployment structure and operating mode of redis. In a real production environment, we basically will not use single-node redis to provide services, at least the sentinel or cluster mode of a master-slave structure to ensure the reliability of the redis service. This article will explain in detail the master-slave synchronization mechanism of redis. [Related recommendations: Redis Video Tutorial]

1. Redis master-slave has two structural models:

1.1 Master-slave replication

The replication structure of one master and N slaves has only one level of replication relationship, which is also the most commonly used form. Usually redis that builds sentry or cluster structures adopts this replication structure, which can be passed through a The replication relationship between level-slave nodes ensures service availability and enables master-slave switching under abnormal conditions.

1.2 Cascade replication

The replication relationship of the cascade replication structure can have multiple levels, and the slave node of a master node can be the master node of a subordinate slave node. The application of cascade replication structure is relatively rare. This structure can alleviate the replication pressure of the master node to a certain extent in a structure with multiple slave nodes.

2. Establishment of Redis master-slave relationship

The master-slave synchronization of redis starts with the command SLAVEOF host port. Through this command, the master-slave relationship can be established. The SLAVEOF command is used to dynamically run Redis Modify the behavior of the copy function. By executing the SLAVEOF host port command, you can turn the current server into a slave server of the specified server. If the current server is already a slave server of a master server, executing SLAVEOF host port will cause the current server to stop synchronizing with the old master server, discard the old data set, and start synchronizing with the new master server. In addition, executing the command SLAVEOF NO ONE on a slave server will cause the slave server to turn off the replication function and transition from the slave server back to the master server. The original synchronized data set will not be discarded. Taking advantage of the feature of SLAVEOF NO ONE that the synchronized data set will not be discarded, without setting up a sentinel or cluster, the slave server can be used as the new master server when the master server fails, thus achieving uninterrupted operation.

The following figure shows the process of establishing a master-slave relationship:

Note:

According to the above execution process, there is a point to note. When we execute the slaveof command on a node that already has a master-slave relationship, the existing master-slave relationship will be ended and all data under the node will be cleared. In the production environment, this is A relatively threatening operation. Is there a safer way? When introducing the slaveof command above, we mentioned that you can pass the NO ONE parameter, that is, execute the SLAVEOF NO ONE command. This command will only end the master-slave replication relationship and will not clear the data. It is relatively safer.

3. Data Synchronization

After the master-slave relationship is established, the master-slave data synchronization process will begin. There are three main situations here. The data after the master-slave relationship is just established is fully synchronized. ;The command propagation stage after the initial synchronization is completed; the synchronization method selection after the master-slave relationship is abnormally interrupted and reconnected. There are two scenarios of full and incremental synchronization.

3.1 Full synchronization

• When the slave node is started or disconnected and reconnected (the reconnection does not meet the incremental synchronization conditions), the SYNC command will be sent to the master database .

• After the master node receives the SYNC command, it will start saving snapshots in the background (i.e. RDB persistence, during master-slave replication, RDB will be triggered unconditionally), and will receive the The commands are cached.

• After the master node completes RDB persistence, it sends snapshot RDB files to all slave nodes, and continues to record the executed write commands during the period of sending the snapshot.

• After the slave node receives the snapshot file, it discards all old data (all data will be cleared) and loads the received snapshot.

• After the master node snapshot is sent and the slave node loads the snapshot, the master node starts to send the write command in the buffer to the slave node.

• The slave node completes loading the snapshot, starts receiving command requests, and executes write commands from the main database buffer. (Initialization completed from database)

• Every time the master node executes a write command, it will send the same write command to the slave node, and the slave node receives and executes the received write command. (Command propagation operation, operation after slave node initialization is completed)

The full synchronization process is as follows:

In redis2.8 Previously, the slave node used full synchronization whether it was initialized or disconnected and reconnected. In versions after 2.8, the PSYNC command was introduced to determine whether to use incremental synchronization after the slave node was disconnected and reconnected.

3.2 Incremental synchronization

PSYNC has full data resynchronization and incremental synchronization modes.

• Full resynchronization: It is basically the same as the old version of replication, and can be understood as "full" replication.

• Partial resynchronization: When the slave is disconnected and reconnected, during the command propagation phase, you only need to send the commands executed during the period of disconnection from the master to the slave. It is understandable. For "incremental" copying.

There are three important concepts in the execution process of PSYNC: runid, offset (copy offset) and copy backlog buffer.

1.runid

Each Redis server will have an ID that indicates its identity. The ID sent in PSYNC refers to the ID of the previously connected Master. If this ID is not saved, the PSYNC command will use the form "PSYNC? -1" to send it to the Master, indicating that full copy is required.

2.offset (replication offset)

Both the Master and Slave in master-slave replication will each maintain an offset. After the Master successfully sends the N-byte command, the offset in the Master will be increased by N. After the Slave receives the N-byte command, the Slave will also increase the offset in the Slave by N. If the status of Master and Slave are consistent, their offsets should also be consistent.

3. Copy backlog buffer

The copy backlog buffer is a fixed-length circular backlog queue (FIFO queue) maintained by the Master. Its function is to cache the Propagate the command. When the Master propagates commands, it not only sends the commands to all Slaves, but also writes the commands into the replication backlog buffer. The difference between the execution process of PSYNC and SYNC is that when salve is connected, it is judged whether full synchronization is required. The logical process of full synchronization is the same as that of SYNC. The execution steps of PSYNC are as follows:

• The client sends the SLAVEOF command to the server, that is, when the slave initiates a connection request to the master, the slave determines whether it is the first connection based on whether it saves the Master runid.

• If it is the first synchronization, the PSYNC ? -1 command will be sent to the Master for complete synchronization; if it is a reconnection, the PSYNC runid offset command will be sent to the Master (the runid is the master's Identity ID, offset is the global migration amount of the slave node synchronization command).

• After the Master receives the PSYNC command, it first determines whether the runid is consistent with the id of the local machine. If it is consistent, it will again determine whether the offset offset is different from the offset of the local machine. The replication backlog buffer size is exceeded. If not, CONTINUE is sent to the Slave. At this time, the Slave only needs to wait for the Master to return the commands lost during the loss of connection. If the runid is inconsistent with the local ID or the offset gap exceeds the copy backlog buffer size, then FULLRESYNC runid offset will be returned, and the slave will save the runid and perform full synchronization.

When the master node propagates commands, the master database will pass each write command to the slave database, and at the same time, it will store the write command in the backlog queue and record the current storage in the backlog queue. The global offset of the command. When the salve reconnects, the master will find the commands executed during the disconnection period in the ring backlog queue based on the offset passed from the slave node, and synchronize them to the salve node to achieve incremental synchronization results.

PSYNC execution process is as shown below:

# From the above PSYNC execution process, we can see that when the slave node is disconnected and reconnected, it is judged whether to use incremental synchronization. The core is whether the difference between the offset offset of the slave and the offset of the master exceeds the copy backlog buffer size. Then this size is configured by the following parameters. The replication backlog buffer is essentially a fixed-length circular queue. By default, the size of the backlog queue is 1MB. The queue size can be set through the configuration file: Set the replication backlog buffer size. The larger the backlog queue, the greater the master-slave database is allowed to be disconnected. The longer the time

repl-backlog-size 1mb

Redis also provides how long it takes to release the ring queue when no slave needs to be synchronized. The default is one hour. When there is no salve connection, how often the replication backlog buffer is released

repl-backlog-ttl 3600

四、主从复制策略

Redis采用了乐观复制的策略，也就是在一定程度内容忍主从数据库的内容不一致，但是保持主从数据库数据的最终一致性。具体来说，Redis在主从复制的过程中，本身就是异步的，在主从数据库执行完客户端请求后会立即将结果返回给客户端，并异步的将命令同步给从数据库，但是这里并不会等待从数据库完全同步之后，再返回客户端。这一特性虽然保证了主从复制期间性能不受影响，但是也会产生一个数据不一致的时间窗口，如果在这个时间窗口期间网络突然断开连接，就会导致两者数据不一致。如果不在配置文件中添加其他策略，那就默认会采用这种方式。为了防止主从不一致不可控，redis提供了以下两个参数来做约束：

min-slaves-to-write 3
min-slaves-max-lag 10

当slave数量小于min-slaves-to-write，且延迟小于等于min-slaves-max-lag时，master停止写入操作。

还有一个参数也会影响主从之间的延时：

repl-disable-tcp-nodelay：

设置成yes，则redis会合并小的TCP包从而节省带宽，但会增加同步延迟，造成master与slave数据不一致。设置成no，则redis master会立即发送同步数据，几乎没有延迟。

Redis的主从同步无论那种场景可以抽象为以下七个步骤：

1.建立socket连接

从服务器根据设置的套接字创建连向主服务器的套接字连接，主服务器接收从服务器的套接字连接之后，为该套接字创建响应的客户端状态，并将此时的从服务器看做是主服务器的客户端，也就是该从服务器同时具备服务器与客户端两个身份。

2.发送PING命令

PING命令主要有两种作用：虽然建立了套接字连接，但是还未使用过，通过发送PING命令检查套接字的读写状态是否正常；通过发送PING命令检查主服务器能否正常处理命令请求，能处理主服务器回复PONG。

3.身份验证

从服务器接收到主服务器返回的“PONG”回复，接下来就需要考虑身份验证的事。如果从服务器设置了masterauth选项，那么进行身份验证，如果从服务器没有设置masterauth选项，那么不进行身份验证。

4.发送端口信息

在身份验证步骤之后，从服务器将执行命令REPLCONF listening-port ，向主服务器发送从服务器的监听端口号。

5.数据同步

从服务器向主服务器发送SYNC命令、PSYNC命令，执行同步操作。

6.命令传播

主从服务器就会进入命令传播阶段，主服务器只要将自己执行的写命令发送给从服务器，而从服务器只要一直执行并接收主服务器发来的写命令。

五、告一段落

本篇详细介绍了redis主从同步机制，不同场景下同步策略的选择，这也是redis高可用的基石。在此基础上，下一篇将对redis高可用的实现来进行分析讲解。

更多编程相关知识，请访问：编程视频！！

The above is the detailed content of In-depth understanding of the master-slave synchronization mechanism in Redis. For more information, please follow other related articles on the PHP Chinese website!