A detailed explanation of high availability and persistence in redis

This article will talk about high availability and persistence in redis, and look at the persistence function and two methods of Redis (RDB and AOF). I hope it will be helpful to you!

1. Redis High Availability

1. Redis High Availability Overview

In web servers, high availability means that the server can operate normally The access time is measured by how long it takes to provide normal services (99.9%, 99.99%, 99.999%, etc.). [Related recommendations: Redis Video Tutorial]

 But in the context of Redis, the meaning of high availability seems to be broader, in addition to ensuring the provision of normal services (such as master-slave separation, rapid disaster recovery technology), it is also necessary to consider the expansion of data capacity, data security and no loss, etc.

2. Redis high availability strategy

In Redis, the technologies to achieve high availability mainly include persistence, master-slave separation, sentinels and clusters.

High availability strategy	Description
Persistence	Persistence is the simplest high-availability method (sometimes not even classified as a high-availability method). Its main function is data backup, that is, storing data on the hard disk to ensure that the data will not be lost due to process exit.
Master-slave replication	Master-slave replication is the basis of high-availability Redis. Both Sentinel and Cluster achieve high availability based on master-slave replication. Master-slave replication mainly implements multi-machine backup of data, as well as load balancing and simple fault recovery for read operations. Defects: Failure recovery cannot be automated, write operations cannot be load balanced, and storage capacity is limited by a single machine.
Sentinel	Based on master-slave replication, Sentinel implements automated fault recovery. Disadvantages: Write operations cannot be load balanced, and storage capacity is limited by a single machine.
Cluster	Through clustering, Redis solves the problem that write operations cannot be load-balanced and storage capacity is limited by a single machine, achieving a relatively complete high-availability solution.

二、Redis 持久化

1. Redis 持久化的功能

  Redis是内存数据库，数据都是存储在内存中，为了避免服务器断电等原因导致Redis进程异常退出后数据的永久丢失，需要定期将Redis中的数据以某种形式（数据或命令）从内存保存到硬盘；当下次Redis重启时，利用持久化文件实现数据恢复。除此之外，为了进行灾难备份，可以将持久化文件拷贝到一个远程位置。

2. Redis 持久化的两种方式

• RDB持久化
  原理是将Redis在内存中的数据库记录定时保存到磁盘上。
• AOF持久化（append only file）
  原理是将Redis的操作日志以追加的方式写入文件，类似于MySQL的binlog。
  由于AOF持久化的实时性更好，即当进程意外退出时丢失的数据更少，因此AOF是目前主流的持久化方式，不过RDB持久化仍然有其用武之地。

3. RDB 持久化

  RDB持久化是指在指定的时间间隔内将内存中当前进程中的数据生成快照保存到硬盘（因此也称为快照持久化），用二进制压缩存储，保存的文件后缀是rdb；当Redis重新启动时，可以读取快照文件恢复数据。

3.1 触发条件

RDB持久化的触发分为手动触发和自动触发两种。

3.1.1 手动触发

• save命令和bgsave命令都可以生成RDB文件。
• save命令会阻塞Redis服务器进程，直到RDB文件创建完毕为止，在Redis服务器阻塞期间，服务器不能处理任何命令请求。
• bgsave命令会fork()一个子进程，由子进程来负责创建RDB文件，父进程（即Redis主进程）则继续处理请求。
• bgsave命令执行过程中，只有fork子进程时会阻塞服务器，而对于save命令，整个过程都会阻塞服务器，因此save已基本被废弃，线上环境要杜绝save的使用。

3.1.2 自动触发

• 在自动触发RDB持久化时，Redis也会选择bgsave而不是save来进行持久化。

3.2 配置方式

• 通过修改配置文件进行设定：save m n
• 自动触发最常见的情况是在配置文件中通过save m n，指定当m秒内发生n次变化时，会触发bgsave。

[root@localhost ~]# vim /etc/redis/6379.conf ##219行，以下三个save条件满足任意一个时，都会引起bgsave的调用save 900 1	##当时间到900秒时，如果redis数据发生了至少1次变化，则执行bgsavesave 300 10	##当时间到300秒时，如果redis数据发生了至少10次变化，则执行bgsavesave 60 10000	##当时间到60秒时，如果redis数据发生了至少10000次变化，则执行bgsave##254行，指定RDB文件名dbfilename dump.rdb##264行，指定RDB文件和AOF文件所在目录dir /var/lib/redis/6379##242行，是否开启RDB文件压缩rdbcompression yes

3.3 其他自动触发机制

除了save m n以外，还有一些其他情况会触发bgsave:

• 在主从复制场景下，如果从节点执行全量复制操作，则主节点会执行bgsave命令，并将rdb文件发送给从节点。
• 执行shutdown命令时，自动执行rdb持久化。

3.4 执行流程

• Redis父进程首先判断：当前是否在执行save，或bgsave/bgrewriteaof的子进程，如果在执行则bgsave命令直接返回。bgsave/bgrewriteaof的子进程不能同时执行，主要是基于性能方面的考虑；两个并发的子进程同时执行大量的磁盘写操作，可能引起严重的性能问题。
• 父进程执行fork操作创建子进程，这个过程中父进程是阻塞的，Redis不能执行来自客户端的任何命令。
• 父进程fork后，bgsave命令返回“Background saving started”信息并不再阻塞父进程，并可以响应其他命令
• 子进程创建RDB文件，根据父进程内存快照生成临时快照文件，完成后对原有文件进行原子替换
• 子进程发送信号给父进程表示完成，父进程更新统计信息

3.5 启动时加载

  RDB文件的载入工作是在服务器启动时自动执行的，并没有专门的命令。但是由于AOF的优先级更高，因此当AOF开启时，Redis会优先载入AOF文件来恢复数据；只有当AOF关闭时，才会在Redis服务器启动时检测RDB文件，并自动载入。服务器载入RDB文件期间处于阻塞状态，直到载入完成为止。
  Redis载入RDB文件时，会对RDB文件进行校验，如果文件损坏，则日志中会打印错误，Redis启动失败。

4. AOF 持久化

  RDB持久化是将进程数据写入文件，而AOF持久化则是将Redis执行的每次写、删除命令记录到单独的日志文件中，查询操作不会记录；当Redis重启时再次执行AOF文件中的命令来恢复数据。
  与RDB相比，AOF的实时性更好，因此已成为主流的持久化方案。

4.1 开启 AOF

Redis服务器默认开启RDB，关闭AOF；要开启AOF，需要在配置文件中配置

[root@localhost ~]# vim /etc/redis/6379.conf ##700行，修改，开启AOFappendonly yes##704行，指定AOF文件名称appendfilename "appendonly.aof"##796行，是否忽略最后一条可能存在问题的指令aof-load-truncated yes[root@localhost ~]# /etc/init.d/redis_6379 restartStopping ...
Redis stopped
Starting Redis server...

4.2 执行流程

由于需要记录Redis的每条写命令，因此AOF不需要触发，下面介绍AOF的执行流程。

AOF的执行流程包括：

• Command append (append): append the Redis write command to the buffer aof_buf;
• File writing (write) and file synchronization (sync): add aof_buf according to different synchronization strategies The content is synchronized to the hard disk;
• File rewrite (rewrite): Rewrite the AOF file regularly to achieve the purpose of compression.

4.2.1 Command append (append)

Redis first appends the command to the buffer instead of writing it directly to the file, mainly to avoid directly writing a command every time Writing to the hard disk causes hard disk IO to become the bottleneck of Redis load.

The format appended by the command is the protocol format requested by the Redis command. It is a plain text format and has the advantages of good compatibility, strong readability, easy processing, simple operation, and avoidance of secondary overhead. In the AOF file, except for the select command used to specify the database (such as select 0 to select database No. 0), which is added by Redis, the others are write commands sent by the client.

4.2.2 File writing (write) and file synchronization (sync)

Redis provides a variety of AOF buffer synchronization file strategies, which involve the write and fsync functions of the operating system. , the description is as follows:
In order to improve the efficiency of file writing, in modern operating systems, when the user calls the write function to write data to the file, the operating system usually temporarily stores the data into a memory buffer. When the buffer The data in the buffer is actually written to the hard disk only after it is filled or exceeds the specified time limit. Although such an operation improves efficiency, it also brings security issues: if the computer shuts down, the data in the memory buffer will be lost; therefore, the system also provides synchronization functions such as fsync and fdatasync, which can force the operating system to immediately transfer the data in the buffer to The data is written to the hard disk to ensure data security.

4.2.3 Three synchronization methods

There are three synchronization methods for the synchronization file strategy of the AOF cache area, which are configured by modifying line 729 of /etc/redis/6379.conf.

4.2.3.1 appendfsync always

Immediately after the command is written to aof_buf, the system fsync operation is called to synchronize to the AOF file. After fsync is completed, the thread returns. In this case, every write command must be synchronized to the AOF file, and hard disk IO becomes a performance bottleneck. Redis can only support about a few hundred TPS writes, seriously reducing the performance of Redis; even if a solid-state drive (SSD) is used, It can only handle tens of thousands of commands per second, and it will greatly reduce the life of the SSD.

4.2.3.2 appendfsync no

After the command is written into aof_buf, the system write operation is called, and fsync synchronization of the AOF file is not performed; the synchronization is loaded by the operating system, and the synchronization period is usually 30 seconds. In this case, the file synchronization time is uncontrollable, and there will be a lot of data accumulated in the buffer, and data security cannot be guaranteed.

4.2.3.3 appendfsync everysec (recommended)

After the command is written to aof_buf, the system write operation is called. After the write is completed, the thread returns: the fsync synchronization file operation is called once per second by a dedicated thread. everysec is a compromise between the two aforementioned strategies, a balance between performance and data security. It is the default configuration of Redis and our recommended configuration.

4.2.4 File rewrite (rewrite)

As time goes by, the Redis server executes more and more write commands, and the AOF file will become larger and larger; an AOF that is too large will The file will not only affect the normal operation of the server, but also cause data recovery to take too long.
File rewriting refers to rewriting the AOF file regularly to reduce the size of the AOF file. It should be noted that AOF rewriting converts the data in the Redis process into write commands and synchronizes them to the new AOF file; no reading or writing operations are performed on the old AOF file.
Another point to note about file rewriting is: for AOF persistence, although file rewriting is highly recommended, it is not necessary; even without file rewriting, data can be persisted and stored in Redis is imported when it starts; therefore, in some realities, automatic file rewriting will be turned off, and then the scheduled task will be executed at a certain time every day.

4.2.4.1 Reasons for having compression function

The reason why file rewriting can compress AOF files is that:

• Expired data is no longer written to the file .
• Invalid commands are no longer written to the file: for example, some data are repeatedly set (set mykey v1, set mykey v2), some data are deleted (set myset v1, del myset), etc.
• Multiple commands can be merged into one: for example, sadd myset v1, sadd myset v2, sadd myset v3 can be merged into sadd myset v1 v2 v3.

It can be seen from the above reasons that since AOF executes fewer commands after rewriting, file rewriting can not only reduce the space occupied by the file, but also speed up the recovery speed.

4.2.4.2 Triggering of file rewriting

File rewriting is divided into manual triggering and automatic triggering:

• Manual triggering: Directly call the bfrewriteaof command. The execution of this command is somewhat similar to bgsave. Both fork processes perform specific work, and they only block when forking.
• Automatic triggering: Automatically execute bgrewriteaof by setting the auto-aof-rewrite-min-size option and the auto-aof-rewrite-percentage option. Only when the two options auto-aof-rewrite-min-size and auto-aof-rewrite-percentage are satisfied at the same time, AOF rewriting, that is, the bgrewriteaof operation will be automatically triggered.

The automatically triggered configuration is located at lines 771 and 772 of /etc/redis/6379.conf

• auto-aof-rewrite-percentage 100
  When the current AOF file size (i.e. aof_current_size) is twice the AOF file size (aof_base_size) when the last log was rewritten, the bgrewriteaof operation occurs
• auto-aof-rewrite-min-size 64mb
  The minimum value for executing the bgrewriteaof command on the current AOF file to avoid frequent bgrewriteaof due to the small file size when starting Redis

4.2.4.3 File rewriting process

The file rewriting process is as follows:

• The Redis parent process first determines whether there is a child process currently executing bgsave/bgrewriteaof; if it exists, the bgrewriteaof command returns directly. If bgsave exists The command will be executed after bgsave is completed.
• The parent process performs a fork operation to create a child process. During this process, the parent process is blocked.
• After the parent process forks, the bgrewriteaof command returns the "Background append only file rewrite started" message and no longer blocks the parent process, and can respond to other commands. All Redis write commands are still written to the AOF buffer and synchronized to the hard disk according to the appendfsync policy to ensure the correctness of the original AOF mechanism.
• Since the fork operation uses copy-on-write technology, the child process can only share the memory data during the fork operation. Since the parent process is still responding to the command, Redis uses the AOF rewrite buffer (aof_rewrite_buf) to save this part of the data to prevent this part of the data from being lost during the generation of the new AOF file. In other words, during the execution of bgrewriteaof, Redis's write command is appended to both aof_buf and aof_rewrite_buf buffers at the same time.
• The child process writes to the new AOF file according to the memory snapshot and the command merging rules.
• After the child process finishes writing the new AOF file, it sends a signal to the parent process, and the parent process updates the statistical information, which can be viewed through info persistence.
• The parent process writes the data in the AOF rewrite buffer to the new AOF file, thus ensuring that the database state saved in the new AOF file is consistent with the current state of the server.
• Replace the old file with the new AOF file and rewrite it into AOF.

Regarding the file rewriting process, there are two points that need special attention:

• The rewriting is performed by the parent process forking the child process
• The write commands executed by Redis during rewriting need to be appended to the new AOF file. For this reason, Redis introduces aof_rewrite_buf cache for example

4.3 Loading at startup

• When AOF is turned on, Redis will load the AOF file first to restore data when it starts; only when AOF is turned off, the RDB file will be loaded to restore data.
• When AOF is turned on but the AOF file does not exist, the RDB file will not be loaded even if it exists.
• When Redis loads an AOF file, it will verify the AOF file. If the file is damaged, an error will be printed in the log and Redis will fail to start. However, if the end of the AOF file is incomplete (a sudden machine downtime can easily cause the end of the file to be incomplete), and the aof_load_truncated parameter is turned on, a warning will be output in the log, and Redis will ignore the end of the AOF file and start successfully. The aof_load_truncated parameter is enabled by default.

5. Advantages and disadvantages of RDB and AOF

RDB persistence

Advantages: RDB files are compact, small in size, and fast in network transmission , suitable for full copy; the recovery speed is much faster than AOF. Of course, one of the most important advantages of RDB is that it has a relatively small impact on performance compared to AOF.
Disadvantages: The well-known disadvantage of RDB files is that the persistence method of data snapshots determines that real-time persistence cannot be achieved. Today, when data is becoming more and more important, a large amount of data loss is often unacceptable. Therefore, AOF persistence has become mainstream. In addition, RDB files need to meet a specific format and have poor compatibility (for example, the old version of Redis is not compatible with the new version of RDB files).
For RDB persistence, on the one hand, the Redis main process will be blocked when bgsave performs a fork operation. On the other hand, the sub-process writing data to the hard disk will also bring IO pressure.

AOF persistence

Corresponding to RDB persistence, the priority of AOF is to support second-level persistence and good compatibility. The disadvantage is that the file is large and the recovery speed is slow. , which has a great impact on performance.

For AOF persistence, the frequency of writing data to the hard disk is greatly increased (second level under everysec policy), the IO pressure is greater, and there may even be additional blocking problems in the AOF.

The rewriting of AOF files is similar to RDB's bgsave, and there will be problems with blocking during fork and IO pressure of the child process. Relatively speaking, because AOF writes data to the hard disk more frequently, it will have a greater impact on the performance of the Redis main process.

Generally speaking, it is recommended to turn off the automatic rewriting function of AOF, and set a scheduled task during the rewriting operation, and perform it in the early morning when the business volume is low, so as to reduce the impact of AOF on the performance of the main process and the read and write pressure of IO .

For more programming-related knowledge, please visit: Introduction to Programming! !

The above is the detailed content of A detailed explanation of high availability and persistence in redis. For more information, please follow other related articles on the PHP Chinese website!