Detailed explanation of the persistence principle of Redis in-depth learning

This article will take you to learn more about Redis, analyze persistence in detail, introduce its working principle, persistence process, etc. I hope it will be helpful to you!

This article will introduce the Redis persistence mechanism from the following aspects:

## Written in front

This article is detailed overall Introducing the two persistence methods of Redis, including working principles, persistence process and practical strategies, as well as some theoretical knowledge behind them. The previous article only introduced RDB persistence, but Redis persistence is a whole and cannot be introduced separately, so it has been reorganized. [Related recommendations: Redis Video Tutorial]

Redis is an in-memory database. All data will be stored in memory. This is directly different from traditional relational databases such as MySQL, Oracle, and SqlServer. Compared with saving data to the hard disk, Redis's reading and writing efficiency is very high. However, saving in memory also has a big flaw. Once the power is cut off or the computer is down, all the contents in the memory database will be lost. In order to make up for this shortcoming, Redis provides the function of persisting memory data to hard disk files and restoring data through backup files, which is the Redis persistence mechanism.

Redis supports two methods of persistence: RDB snapshot and AOF.

RDB persistence

RDB snapshot In official words: RDB persistence solution is a point-to-time snapshot generated for your data set at a specified time interval. . It saves the memory snapshot of all data objects in the Redis database at a certain time in a compressed binary file, which can be used for Redis data backup, transfer and recovery. So far, it is still the official default support solution.

RDB working principle

Since RDB is a point-in-time snapshot of the data set in Redis, let’s first briefly understand how the data objects in Redis are stored and organized in memory.

By default, there are 16 databases in Redis, numbered from 0-15. Each Redis database is represented by a redisDb object, redisDb uses hashtable storage K-V object. To facilitate understanding, I took one of the dbs as an example to draw a schematic diagram of the storage structure of Redis's internal data.

The point-in-time snapshot is the status of each data object in each DB in Redis at a certain moment. First assume that all If the data objects no longer change, we can read these data objects one by one and write them to files according to the data structure relationship in the above figure to achieve Redis persistence. Then, when Redis restarts, the contents of this file are read according to the rules, and then written to the Redis memory to restore to the state of persistence.

Of course, this premise is true when our above assumption is true, otherwise we would have no way to start in the face of a data set that changes all the time. We know that client command processing in Redis is a single-threaded model. If persistence is processed as a command, the data set will definitely be in a static state. In addition, the child process created by the fork() function provided by the operating system can obtain the same memory data as the parent process, which is equivalent to obtaining a copy of the memory data; after the fork is completed, what should the parent process do, and the work of persisting the state is handed over. Just give it to the child process.

Obviously, the first situation is not advisable. Persistent backup will cause the Redis service to be unavailable for a short period of time, which is intolerable for high HA systems. Therefore, the second method is the main practical method of RDB persistence. Since the data of the parent process keeps changing after forking the child process, and the child process is not synchronized with the parent process, RDB persistence cannot guarantee real-time performance; a power outage or downtime after the RDB persistence is completed will result in partial data loss; backup The frequency determines the amount of lost data. Increasing the backup frequency means that the fork process consumes more CPU resources and will also lead to larger disk I/O.

Persistence process

There are two methods to complete RDB persistence in Redis: rdbSave and rdbSaveBackground (in the source code file rdb.c). Let’s briefly talk about the difference between the two:

• rdbSave: It is executed synchronously, and the persistence process will be started immediately after the method is called. Since Redis is a single-threaded model, it will be blocked during the persistence process, and Redis cannot provide external services;
• rdbSaveBackground: It is executed in the background (asynchronously). This method will fork out the child process. The real persistence process is When executed in a child process (calling rdbSave), the main process will continue to provide services;

The triggering of RDB persistence must be inseparable from the above two methods, and the triggering methods are divided into manual and automatic. Manual triggering is easy to understand. It means that we manually initiate persistence backup instructions to the Redis server through the Redis client, and then the Redis server starts to execute the persistence process. The instructions here include save and bgsave. Automatic triggering is a persistence process that Redis automatically triggers when preset conditions are met based on its own operating requirements. The automatically triggered scenarios are as follows (extracted from this article):

• serverCron中save m nThe configuration rules are automatically triggered;
• When the slave node is fully copied, the master node sends the rdb file to the slave node to complete the copy operation, and the master node will start bgsave;
• Executedebug reloadWhen the command reloads redis;
• By default (AOF is not enabled), when the shutdown command is executed, bgsave is automatically executed;

Combined with the source code and reference articles, I have organized the RDB persistence process to help you have an overall understanding, and then explain it in some details.

As you can see from the above figure:

• The automatically triggered RDB persistence is a persistence strategy executed in a sub-process through rdbSaveBackground;
• Manual triggering is triggered by client commands, including save and bgsave commands. The save command is completed by calling the rdbSave method in the Redis command processing thread in a blocking manner.

The automatic triggering process is a complete link, covering rdbSaveBackground, rdbSave, etc. Next, I will use serverCron as an example to analyze the entire process.

save rules and checks

serverCron is a periodic function in Redis, executed every 100 milliseconds. One of its jobs is to determine the current situation based on the save rules in the configuration file. An automatic persistence process is required, and if the conditions are met, an attempt will be made to start persistence. Learn about the implementation of this part.

There are several fields related to RDB persistence in redisServer. I extracted them from the code and looked at them in Chinese and English:

struct redisServer {
    /* 省略其他字段 */ 
    /* RDB persistence */
    long long dirty;                /* Changes to DB from the last save
                                     * 上次持久化后修改key的次数 */
    struct saveparam *saveparams;   /* Save points array for RDB，
                                     * 对应配置文件多个save参数 */
    int saveparamslen;              /* Number of saving points，
                                     * save参数的数量 */
    time_t lastsave;                /* Unix time of last successful save 
                                     * 上次持久化时间*/
    /* 省略其他字段 */
}

/* 对应redis.conf中的save参数 */
struct saveparam {
    time_t seconds;                    /* 统计时间范围 */   
    int changes;                    /* 数据修改次数 */
};

saveparams corresponds to the save rule under redis.conf. The save parameter is the triggering strategy for Redis to trigger automatic backup, seconds is the statistical time (unit: seconds), changes is the number of writes that occurred within the statistical time. save m n means: if there are n writes within m seconds, a snapshot will be triggered, that is, a backup. Multiple groups of save parameters can be configured to meet backup requirements under different conditions. If you need to turn off the automatic backup policy of RDB, you can use save "". The following are descriptions of several configurations:

# 表示900秒（15分钟）内至少有1个key的值发生变化，则执行
save 900 1
# 表示300秒（5分钟）内至少有1个key的值发生变化，则执行
save 300 10
# 表示60秒（1分钟）内至少有10000个key的值发生变化，则执行
save 60 10000
# 该配置将会关闭RDB方式的持久化
save ""

serverCronThe detection code for the RDB save rule is as follows:

int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
    /* 省略其他逻辑 */
    
    /* 如果用户请求进行AOF文件重写时，Redis正在执行RDB持久化，Redis会安排在RDB持久化完成后执行AOF文件重写，
     * 如果aof_rewrite_scheduled为true，说明需要执行用户的请求 */
    /* Check if a background saving or AOF rewrite in progress terminated. */
    if (hasActiveChildProcess() || ldbPendingChildren())
    {
        run_with_period(1000) receiveChildInfo();
        checkChildrenDone();
    } else {
        /* 后台无 saving/rewrite 子进程才会进行，逐个检查每个save规则*/
        for (j = 0; j = sp->changes 
                && server.unixtime-server.lastsave > sp->seconds 
                &&(server.unixtime-server.lastbgsave_try > CONFIG_BGSAVE_RETRY_DELAY || server.lastbgsave_status == C_OK))
            {
                serverLog(LL_NOTICE,"%d changes in %d seconds. Saving...", sp->changes, (int)sp->seconds);
                rdbSaveInfo rsi, *rsiptr;
                rsiptr = rdbPopulateSaveInfo(&rsi);
                /* 执行bgsave过程 */
                rdbSaveBackground(server.rdb_filename,rsiptr);
                break;
            }
        }

        /* 省略：Trigger an AOF rewrite if needed. */
    }
    /* 省略其他逻辑 */
}

If there is no background RDB persistence or AOF replication When writing a process, serverCron will judge whether persistence operations need to be performed based on the above configuration and status. The basis for judgment is whether lastsave and dirty satisfy one of the conditions in the saveparams array. If a condition is matched, the rdbSaveBackground method is called to execute the asynchronous persistence process.

rdbSaveBackground

rdbSaveBackground is an auxiliary method for RDB persistence. Its main job is to fork the child process, and then there are two different execution logics depending on the caller (parent process or child process). .

• If the caller is the parent process, fork out the child process, save the child process information and return directly.
• If the caller is a child process, call rdbSave to execute RDB persistence logic, and exit the child process after persistence is completed.

int rdbSaveBackground(char *filename, rdbSaveInfo *rsi) {
    pid_t childpid;

    if (hasActiveChildProcess()) return C_ERR;

    server.dirty_before_bgsave = server.dirty;
    server.lastbgsave_try = time(NULL);

    // fork子进程
    if ((childpid = redisFork(CHILD_TYPE_RDB)) == 0) {
        int retval;

        /* Child 子进程：修改进程标题 */
        redisSetProcTitle("redis-rdb-bgsave");
        redisSetCpuAffinity(server.bgsave_cpulist);
        // 执行rdb持久化
        retval = rdbSave(filename,rsi);
        if (retval == C_OK) {
            sendChildCOWInfo(CHILD_TYPE_RDB, 1, "RDB");
        }
        // 持久化完成后，退出子进程
        exitFromChild((retval == C_OK) ? 0 : 1);
    } else {
        /* Parent 父进程：记录fork子进程的时间等信息*/
        if (childpid == -1) {
            server.lastbgsave_status = C_ERR;
            serverLog(LL_WARNING,"Can't save in background: fork: %s",
                strerror(errno));
            return C_ERR;
        }
        serverLog(LL_NOTICE,"Background saving started by pid %ld",(long) childpid);
        // 记录子进程开始的时间、类型等。
        server.rdb_save_time_start = time(NULL);
        server.rdb_child_type = RDB_CHILD_TYPE_DISK;
        return C_OK;
    }
    return C_OK; /* unreached */
}

rdbSave is a method that truly performs persistence. It involves a large number of I/O and calculation operations during execution, which is time-consuming and takes up a lot of CPU. It is persisted in the single-threaded model of Redis. The process will continue to occupy thread resources, causing Redis to be unable to provide other services. In order to solve this problem, Redis forks out the child process in rdbSaveBackground, and the child process completes the persistence work, avoiding occupying too many resources of the parent process.

It should be noted that if the memory usage of the parent process is too large, the fork process will be more time-consuming. During this process, the parent process cannot provide external services; in addition, the computer memory usage needs to be comprehensively considered before forking the child process. It will occupy double the memory resources, so you need to ensure that the memory is sufficient. Check the latest_fork_usec option through the info stats command to get the time taken for the latest fork operation.

rdbSave

Redis的rdbSave函数是真正进行RDB持久化的函数，流程、细节贼多，整体流程可以总结为：创建并打开临时文件、Redis内存数据写入临时文件、临时文件写入磁盘、临时文件重命名为正式RDB文件、更新持久化状态信息（dirty、lastsave）。其中“Redis内存数据写入临时文件”最为核心和复杂，写入过程直接体现了RDB文件的文件格式，本着一图胜千言的理念，我按照源码流程绘制了下图。

补充说明一下，上图右下角“遍历当前数据库的键值对并写入”这个环节会根据不同类型的Redis数据类型及底层数据结构采用不同的格式写入到RDB文件中，不再展开了。我觉得大家对整个过程有个直观的理解就好，这对于我们理解Redis内部的运作机制大有裨益。

AOF持久化

上一节我们知道RDB是一种时间点（point-to-time）快照，适合数据备份及灾难恢复，由于工作原理的“先天性缺陷”无法保证实时性持久化，这对于缓存丢失零容忍的系统来说是个硬伤，于是就有了AOF。

AOF工作原理

AOF是Append Only File的缩写，它是Redis的完全持久化策略，从1.1版本开始支持；这里的file存储的是引起Redis数据修改的命令集合（比如：set/hset/del等），这些集合按照Redis Server的处理顺序追加到文件中。当重启Redis时，Redis就可以从头读取AOF中的指令并重放，进而恢复关闭前的数据状态。

AOF持久化默认是关闭的，修改redis.conf以下信息并重启，即可开启AOF持久化功能。

# no-关闭，yes-开启，默认no
appendonly yes
appendfilename appendonly.aof

AOF本质是为了持久化，持久化对象是Redis内每一个key的状态，持久化的目的是为了在Reids发生故障重启后能够恢复至重启前或故障前的状态。相比于RDB，AOF采取的策略是按照执行顺序持久化每一条能够引起Redis中对象状态变更的命令，命令是有序的、有选择的。把aof文件转移至任何一台Redis Server，从头到尾按序重放这些命令即可恢复如初。举个例子：

首先执行指令set number 0，然后随机调用incr number、get number 各5次，最后再执行一次get number ，我们得到的结果肯定是5。

因为在这个过程中，能够引起number状态变更的只有set/incr类型的指令，并且它们执行的先后顺序是已知的，无论执行多少次get都不会影响number的状态。所以，保留所有set/incr命令并持久化至aof文件即可。按照aof的设计原理，aof文件中的内容应该是这样的（这里是假设，实际为RESP协议）：

set number 0
incr number
incr number
incr number
incr number
incr number

最本质的原理用“命令重放”四个字就可以概括。但是，考虑实际生产环境的复杂性及操作系统等方面的限制，Redis所要考虑的工作要比这个例子复杂的多：

• Redis Server启动后，aof文件一直在追加命令，文件会越来越大。文件越大，Redis重启后恢复耗时越久；文件太大，转移工作就越难；不加管理，可能撑爆硬盘。很显然，需要在合适的时机对文件进行精简。例子中的5条incr指令很明显的可以替换为为一条set命令，存在很大的压缩空间。
• 众所周知，文件I/O是操作系统性能的短板，为了提高效率，文件系统设计了一套复杂的缓存机制，Redis操作命令的追加操作只是把数据写入了缓冲区（aof_buf），从缓冲区到写入物理文件在性能与安全之间权衡会有不同的选择。
• 文件压缩即意味着重写，重写时即可依据已有的aof文件做命令整合，也可以先根据当前Redis内数据的状态做快照，再把存储快照过程中的新增的命令做追加。
• aof备份后的文件是为了恢复数据，结合aof文件的格式、完整性等因素，Redis也要设计一套完整的方案做支持。

持久化流程

从流程上来看，AOF的工作原理可以概括为几个步骤：命令追加（append）、文件写入与同步（fsync）、文件重写（rewrite）、重启加载（load），接下来依次了解每个步骤的细节及背后的设计哲学。

Command Append

When the AOF persistence function is turned on, after Redis executes a write command, it will use the protocol format (that is, RESP, the communication protocol for the interaction between the Redis client and the server) Append the executed write command to the end of the AOF buffer maintained by the Redis server. There is only a single-threaded append operation for AOF files, and there are no complex operations such as seek. There is no risk of file damage even if there is a power outage or downtime. In addition, there are many benefits to using the text protocol:

• The text protocol has good compatibility;
• The text protocol is the client’s request command and does not require secondary processing, saving storage and Processing overhead during loading;
• The text protocol is readable and convenient for viewing, modification, etc.

The AOF buffer type is a data structure designed independently by Redis sds. Redis will use different methods according to the type of command (catAppendOnlyGenericCommand, catAppendOnlyExpireAtCommand, etc.) processes the command content and finally writes it to the buffer.

It should be noted that if AOF rewriting is in progress when commands are appended, these commands will also be appended to the rewrite buffer (aof_rewrite_buffer).

File writing and synchronization

The writing and synchronization of AOF files cannot be separated from the support of the operating system. Before starting to introduce, we need to add some knowledge about Linux I/O buffers. The I/O performance of the hard disk is poor, and the file reading and writing speed is far inferior to the processing speed of the CPU. If you wait for data to be written to the hard disk every time you write a file, it will lower the overall performance of the operating system. In order to solve this problem, the operating system provides a delayed write mechanism to improve the I/O performance of the hard disk.

Traditional UNIX implementations have a buffer cache or page cache in the kernel, and most disk I/O occurs through buffering. When writing data to a file, the kernel usually copies the data to one of the buffers first. If the buffer is not full yet, it does not queue it into the output queue, but waits for it to fill up or when the kernel needs to When the buffer is reused to store other disk block data, the buffer is queued into the output queue, and then the actual I/O operation is performed only when it reaches the head of the queue. This output method is called delayed writing.

Delayed writing reduces the number of disk reads and writes, but it also reduces the update speed of file content, so that the data to be written to the file is not written to the disk for a period of time. When a system failure occurs, this delay can cause file updates to be lost. In order to ensure the consistency of the actual file system on the disk and the content in the buffer cache, the UNIX system provides three functions: sync, fsync and fdatasync to provide support for forced writing to the hard disk.

Redis will call the function flushAppendOnlyFile before each event rotation ends (beforeSleep), flushAppendOnlyFile will flush the AOF buffer (aof_buf ) is written to the kernel buffer, and the strategy used to write the data in the kernel buffer to disk is determined based on the appendfsync configuration, that is, calling fsync(). This configuration has three optional options always, no, everysec, the details are as follows:

• always: Called every timefsync() is the strategy with the highest security and the worst performance.
• no: fsync() will not be called. Best performance, worst security.
• everysec: Call fsync() only when synchronization conditions are met. This is the officially recommended synchronization strategy and is also the default configuration. It takes into account both performance and data security. In theory, only 1 second of data will be lost in the event of a sudden system shutdown.

Note: The strategy introduced above is affected by the configuration item no-appendfsync-on-rewrite. Its function is to inform Redis: whether during AOF file rewriting Disable calling fsync(), the default is no.

If appendfsync is set to always or everysec, BGSAVE or is ongoing in the background BGREWRITEAOF consumes too much disk I/O. Under certain Linux system configurations, Redis's call to fsync() may block for a long time. However, this problem has not been fixed yet, because the synchronous write operation will be blocked even if fsync() is executed in a different thread.

To mitigate this problem, this option can be used to prevent fsync() from being called in the main process while doing BGSAVE or BGREWRITEAOF.

• 设置为yes意味着，如果子进程正在进行BGSAVE或BGREWRITEAOF，AOF的持久化能力就与appendfsync设置为no有着相同的效果。最糟糕的情况下，这可能会导致30秒的缓存数据丢失。
• 如果你的系统有上面描述的延迟问题，就把这个选项设置为yes，否则保持为no。

文件重写

如前面提到的，Redis长时间运行，命令不断写入AOF，文件会越来越大，不加控制可能影响宿主机的安全。

为了解决AOF文件体积问题，Redis引入了AOF文件重写功能，它会根据Redis内数据对象的最新状态生成新的AOF文件，新旧文件对应的数据状态一致，但是新文件会具有较小的体积。重写既减少了AOF文件对磁盘空间的占用，又可以提高Redis重启时数据恢复的速度。还是下面这个例子，旧文件中的6条命令等同于新文件中的1条命令，压缩效果显而易见。

我们说，AOF文件太大时会触发AOF文件重写，那到底是多大呢？有哪些情况会触发重写操作呢？
**
与RDB方式一样，AOF文件重写既可以手动触发，也会自动触发。手动触发直接调用bgrewriteaof命令，如果当时无子进程执行会立刻执行，否则安排在子进程结束后执行。自动触发由Redis的周期性方法serverCron检查在满足一定条件时触发。先了解两个配置项：

• auto-aof-rewrite-percentage：代表当前AOF文件大小（aof_current_size）和上一次重写后AOF文件大小（aof_base_size）相比，增长的比例。
• auto-aof-rewrite-min-size：表示运行BGREWRITEAOF时AOF文件占用空间最小值，默认为64MB；

Redis启动时把aof_base_size初始化为当时aof文件的大小，Redis运行过程中，当AOF文件重写操作完成时，会对其进行更新；aof_current_size为serverCron执行时AOF文件的实时大小。当满足以下两个条件时，AOF文件重写就会触发：

增长比例：(aof_current_size - aof_base_size) / aof_base_size > auto-aof-rewrite-percentage
文件大小：aof_current_size > auto-aof-rewrite-min-size

手动触发与自动触发的代码如下，同样在周期性方法serverCron中：

int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
    /* 省略其他逻辑 */
    
    /* 如果用户请求进行AOF文件重写时，Redis正在执行RDB持久化，Redis会安排在RDB持久化完成后执行AOF文件重写，
     * 如果aof_rewrite_scheduled为true，说明需要执行用户的请求 */
    if (!hasActiveChildProcess() &&
        server.aof_rewrite_scheduled)
    {
        rewriteAppendOnlyFileBackground();
    }

    /* Check if a background saving or AOF rewrite in progress terminated. */
    if (hasActiveChildProcess() || ldbPendingChildren())
    {
        run_with_period(1000) receiveChildInfo();
        checkChildrenDone();
    } else {
        /* 省略rdb持久化条件检查 */

        /* AOF重写条件检查：aof开启、无子进程运行、增长百分比已设置、当前文件大小超过阈值 */
        if (server.aof_state == AOF_ON &&
            !hasActiveChildProcess() &&
            server.aof_rewrite_perc &&
            server.aof_current_size > server.aof_rewrite_min_size)
        {
            long long base = server.aof_rewrite_base_size ?
                server.aof_rewrite_base_size : 1;
            /* 计算增长百分比 */
            long long growth = (server.aof_current_size*100/base) - 100;
            if (growth >= server.aof_rewrite_perc) {
                serverLog(LL_NOTICE,"Starting automatic rewriting of AOF on %lld%% growth",growth);
                rewriteAppendOnlyFileBackground();
            }
        }
    }
    /**/
}

AOF文件重写的流程是什么？听说Redis支持混合持久化，对AOF文件重写有什么影响？

从4.0版本开始，Redis在AOF模式中引入了混合持久化方案，即：纯AOF方式、RDB+AOF方式，这一策略由配置参数aof-use-rdb-preamble（使用RDB作为AOF文件的前半段）控制，默认关闭(no)，设置为yes可开启。所以，在AOF重写过程中文件的写入会有两种不同的方式。当aof-use-rdb-preamble的值是：

• no：按照AOF格式写入命令，与4.0前版本无差别；
• yes：先按照RDB格式写入数据状态，然后把重写期间AOF缓冲区的内容以AOF格式写入，文件前半部分为RDB格式，后半部分为AOF格式。

结合源码（6.0版本，源码太多这里不贴出，可参考aof.c）及参考资料，绘制AOF重写（BGREWRITEAOF）流程图：

结合上图，总结一下AOF文件重写的流程：

• rewriteAppendOnlyFileBackground开始执行，检查是否有正在进行的AOF重写或RDB持久化子进程：如果有，则退出该流程；如果没有，则继续创建接下来父子进程间数据传输的通信管道。执行fork()操作，成功后父子进程分别执行不同的流程。

• 父进程：

  • 记录子进程信息（pid）、时间戳等；
  • 继续响应其他客户端请求；
  • 收集AOF重写期间的命令，追加至aof_rewrite_buffer；
  • 等待并向子进程同步aof_rewrite_buffer的内容；

• 子进程：

  • Modify the current process name, create the temporary file required for rewriting, and call the rewriteAppendOnlyFile function;
  • According to the aof-use-rdb-preamble configuration, use RDB or AOF Write the first half in AOF mode and synchronize it to the hard disk;
  • Receive the incremental AOF command from the parent process, write the second half in AOF mode, and synchronize it to the hard disk;
  • Rename the AOF file , the child process exits.

Data loading

After Redis is started, the data loading work is performed through the loadDataFromDisk function. It should be noted here that although the persistence method can be AOF, RDB, or both, a choice must be made when loading data. Loading the two methods separately will cause chaos.

Theoretically, AOF persistence has better real-time performance than RDB. When AOF persistence is enabled, Redis gives priority to AOF when loading data. Moreover, after Redis 4.0 version, AOF supports hybrid persistence, and version compatibility needs to be considered when loading AOF files. The Redis data loading process is shown in the figure below:

In AOF mode, the file generated by turning on the hybrid persistence mechanism is "RDB head AOF tail", which is generated when it is not turned on. All files are in AOF format. Considering the compatibility of the two file formats, if Redis finds that the AOF file is an RDB header, it will use the RDB data loading method to read and restore the first half; and then use the AOF method to read and restore the second half. Since the data stored in the AOF format are RESP protocol commands, Redis uses a pseudo client to execute commands to recover the data.

If a downtime occurs during the appending process of the AOF command, the RESP command of the AOF may be incomplete (truncated) due to the technical characteristics of delayed writing. When encountering this situation, Redis will execute different processing strategies according to the configuration item aof-load-truncated. This configuration tells Redis to read the aof file when starting, and what to do if the file is found to be truncated (incomplete):

• #yes: Load as much data as possible and log it Notify the user;
• no: It will crash with a system error and prohibit startup. The user needs to repair the file and then restart.

Summary

Redis provides two persistence options: RDB supports generating point-in-time snapshots for data sets at specific practical intervals; AOF stores each piece of data received by the Redis Server. The write command is persisted to the log, and the data can be restored by replaying the command when Redis is restarted. The log format is RESP protocol, and only the append operation is performed on the log file, so there is no risk of damage. And when the AOF file is too large, the compressed file can be automatically rewritten.

Of course, if you don't need to persist data, you can also disable Redis's persistence function, but this is not the case in most cases. In fact, we may sometimes use RDB and AOF at the same time. The most important thing is that we understand the difference between the two in order to use them reasonably.

RDB vs AOF

RDB advantages

• RDB is a compact, compressed binary file that represents a snapshot of Redis data at a certain point in time. It is very suitable for backup, Full copy and other scenarios.
• RDB is very friendly to disaster recovery and data migration. RDB files can be moved to any place needed and reloaded.
• RDB is a memory snapshot of Redis data. The data recovery speed is faster and has higher performance than AOF command replay.

RDB Disadvantages

• RDB method cannot achieve real-time or second-level persistence. Because the persistence process is completed by the child process after forking the child process, the memory of the child process is only a snapshot of the data of the parent process at the moment of the fork operation. After the fork operation, the parent process continues to serve the outside world, and the internal data changes all the time. The data is no longer updated, and there are always differences between the two, so real-time performance cannot be achieved.
• The fork operation during the RDB persistence process will double the memory usage, and the more data the parent process has, the longer the fork process will be.
• High concurrency of Redis requests may frequently hit the save rule, causing the frequency of fork operations and persistent backups to be uncontrollable;
• RDB files have file format requirements, and different versions of Redis will change the file format. Adjustments are made, and there is a problem that the old version is not compatible with the new version.

AOF advantages

• AOF persistence has better real-time performance. We can choose three different methods (appendfsync): no, every second, always, every As the default strategy, second has the best performance. In extreme cases, one second of data may be lost.
• AOF files only have append operations, no complicated seek and other file operations, and no risk of damage. Even if the last written data is truncated, it can be easily repaired using the redis-check-aof tool;
• When the AOF file becomes larger, Redis can automatically rewrite it in the background. During the rewriting process, the old file will continue to be written. After the rewriting is completed, the new file will become smaller, and the incremental command during the rewriting process will also be appended to the new file.
• The AOF file contains all operation commands for data in Redis in a way that is easy to understand and parse. Even if all data is accidentally cleared by mistake, as long as the AOF file is not rewritten, we can retrieve all data by removing the last command.
• AOF already supports hybrid persistence, the file size can be effectively controlled, and the efficiency of data loading is improved.

AOF Disadvantages

• For the same data collection, AOF files are usually larger than RDB files;
• Under a specific fsync policy, AOF will Slightly slower than RDB. Generally speaking, the performance of fsync_every_second is still very high, and the performance of fsync_no is comparable to RDB. But under huge write pressure, RDB can provide the greatest low-latency guarantee.
• On AOF, Redis once encountered some rare bugs that are almost impossible to encounter on RDB. Some special instructions (such as BRPOPLPUSH) cause the reloaded data to be inconsistent with that before persistence. Redis officials once tested under the same conditions, but the problem could not be reproduced.

Usage Suggestions

After understanding the working principles, execution processes, advantages and disadvantages of the two persistence methods of RDB and AOF, let’s think about how to weigh the pros and cons in actual scenarios. , use two persistence methods reasonably. If you only use Redis as a caching tool, all data can be reconstructed based on the persistent database. You can turn off the persistence function and do protection work such as preheating, cache penetration, breakdown, and avalanche.

Generally, Redis will take on more work, such as distributed locks, rankings, registration centers, etc. The persistence function will play a greater role in disaster recovery and data migration. It is recommended to follow several principles:

• Do not use Redis as a database. All data can be automatically reconstructed by the application service as much as possible.
• Use version 4.0 or above of Redis and use the AOF RDB hybrid persistence function.
• Reasonably plan the maximum memory occupied by Redis to prevent insufficient resources during AOF rewriting or save.
• Avoid deploying multiple instances on a single machine.
• The production environment is mostly deployed in clusters. The persistence capability can be enabled on the slave so that the master can better provide external write services.
• Backup files should be automatically uploaded to off-site computer rooms or cloud storage to prepare for disaster backup.

About fork()

Through the above analysis, we all know that RDB snapshots and AOF rewriting require fork. This is a heavyweight operation that will cause damage to Redis. block. Therefore, in order not to affect the response of the Redis main process, we need to reduce blocking as much as possible.

• Reduce the frequency of fork, for example, you can manually trigger RDB to generate snapshots and AOF rewrite;
• Control the maximum memory usage of Redis to prevent fork from taking too long;
• Use higher performance hardware;
• Properly configure the Linux memory allocation strategy to avoid fork failure due to insufficient physical memory.

For more programming-related knowledge, please visit: Introduction to Programming! !

The above is the detailed content of Detailed explanation of the persistence principle of Redis in-depth learning. For more information, please follow other related articles on the PHP Chinese website!