Example analysis of AOF persistence in Redis

1. Introduction to AOF

RDB is a persistence method that can record the status of the database by storing key-value pairs in the database. AOF is a persistence method that saves the database state by recording write commands executed by the Redis server.

For example, for the following command:

　

The RDB persistence method is to save the three key-value pairs of str1, str2, str3 into the RDB file, and AOF persistence saves the executed set, sadd, and lpush commands to the AOF file.

2. AOF configuration

Under APPEND ONLY MODE in the redis.conf configuration file:

　

　①、appendonly: The default value is no, which means that redis uses rdb persistence by default. If you want to enable AOF persistence, you need to change appendonly to yes.

　②、appendfilename：aof file name, the default is "appendonly.aof"

　③、appendfsync：configuration of aof persistence strategy;

no means not to execute fsync, and the operating system ensures that the data is synchronized to the disk, which is the fastest, but not very safe;

always means that fsync is executed for every write to ensure data synchronization to disk, the efficiency is very low;

The option "everysec" that executes fsync once per second may cause data to be lost within this 1 second. Usually choose everysec to balance security and efficiency.

　④. no-appendfsync-on-rewrite: When aof is rewritten or written to an rdb file, a large amount of IO will be performed. At this time, for everysec and always aof modes, executing fsync will cause blocking. For too long, the no-appendfsync-on-rewrite field is set to no by default. If the application has high latency requirements, this field can be set to yes, otherwise it is set to no, which is a safer choice for the persistence feature. Setting it to yes means that new write operations will not be fsyncd during rewrite, and will be temporarily stored in the memory, and will be written after rewrite is completed. The default is no, and yes is recommended. The default fsync policy of Linux is 30 seconds. 30 seconds of data may be lost. The default value is no.

The default value of auto-aof-rewrite-percentage is 100. aof automatic rewrite configuration, when the current aof file size exceeds the percentage of the last rewritten aof file size, it will be rewritten, that is, when the aof file grows to a certain size, Redis can call bgrewriteaof to rewrite the log file . When the size of the AOF file reaches twice the size of the last log rewrite (set to 100), a new log rewrite process will be automatically started.

auto-aof-rewrite-min-size is set to 64MB. To avoid the need to rewrite when the agreed percentage is reached, you can set a minimum aof file size.

　⑦. aof-load-truncated: The aof file may be incomplete at the end. When redis starts, the data of the aof file is loaded into the memory. Restart may occur after the host operating system where redis is located is down, especially if the ext4 file system does not add the data=ordered option. This phenomenon occurs. Redis downtime or abnormal termination will not cause the tail to be incomplete. You can choose to let redis exit. , or import as much data as possible. Once the yes option is selected, a log will be automatically sent to the client and loaded when the truncated aof file is imported. If the answer is no, the user needs to manually run the redis-check-aof command to repair the AOF file. The default value is yes.

3. Turn on AOF

Change the appendonly configuration of redis.conf to yes.

　The location where AOF saves files is the same as the location where RDB saves files. They are configured through the dir of the redis.conf configuration file:

　

　You can pass config get The dir command gets the saved path.

4. AOF file recovery

After restarting Redis, the AOF file will be loaded.

Exception repair command: redis-check-aof --fix to repair

5. AOF rewriting

Since AOF persistence means Redis continuously records write commands to AOF In the file, as Redis continues to progress, the AOF file will become larger and larger. The larger the file, the larger the server memory occupied and the longer the AOF recovery requires. In order to solve this problem, Redis has added a new rewriting mechanism. When the size of the AOF file exceeds the set threshold, Redis will start the content compression of the AOF file, retaining only the minimum instruction set that can recover the data. You can use the command bgrewriteaof to rewrite it.

For example, for the following commands:

　

If the AOF file is not rewritten, the AOF file will save four SADD commands. If AOF rewriting is used, then Only the following command will be retained in the AOF file:

1	sadd animals "dog" "tiger " "panda" "lion" "cat"

##　

That is to say, AOF file rewriting does not reorganize the original file, but directly reads the existing key-value pair of the server, and then uses one command to replace the multiple commands that previously recorded the key-value pair to generate a Then replace the original AOF file with the new file.

AOF file rewrite triggering mechanism: through auto-aof-rewrite-percentage in the redis.conf configuration file: the default value is 100, and auto-aof-rewrite-min-size: 64mb configuration , that is to say, by default Redis will record the AOF size when it was last rewritten.

The default configuration is triggered when the AOF file size is twice the size after the last rewrite and the file is larger than 64M.

　Let me mention it again, we know that Redis is a single-threaded job. If it takes a long time to rewrite AOF, then during the rewriting of AOF, Redis will not be able to work for a long time. Dealing with other commands, this is obviously intolerable. In order to overcome this problem, Redis's solution is to put the AOF rewriting program into a subroutine. This has two benefits:

　① During the AOF rewriting of the child process, the server process (parent process) can Continue processing other commands.

Using child processes instead of threads can avoid using locks and ensure data security, because the child process has a copy of the data of the parent process.

Using sub-processes solves the above problem, but new problems also arise: because the server process is still processing other commands during the AOF rewriting process of the sub-process, this new command may also perform operations on the database. The modification operation makes the current database status and the rewritten AOF file status inconsistent.

In order to solve the problem of inconsistent data status, the Redis server sets up an AOF rewrite buffer. This buffer is used after the child process is created. When the Redis server executes a write command, it will This write command is also sent to the AOF rewrite buffer. When the child process completes the AOF rewriting, it will send a signal to the parent process. After the parent process receives the signal, it will call a function to write the contents of the AOF rewriting buffer to the new AOF file.

This minimizes the impact of AOF rewriting on the server.

6. Advantages and Disadvantages of AOF

Advantages:

①. The AOF persistence method provides a variety of synchronization frequencies, even if the default synchronization frequency is used to synchronize every second At one time, Redis only loses 1 second of data at most.

　②. AOF files are constructed using Redis command append. Therefore, even if Redis can only write command fragments to the AOF file, it is easy to correct the AOF file using the redis-check-aof tool.

The format of AOF files is easy to read, which allows users to process them more flexibly. For example, if we accidentally use the FLUSHALL command by mistake, before rewriting is in progress, we can manually remove the last FLUSHALL command and then use AOF to recover the data.

Disadvantages:

① For Redis with the same data, AOF files are usually larger than RDF files.

AOF's default synchronization frequency is once per second, although it provides a variety of synchronization frequency options, but the performance is still high. When Redis load is high, RDB provides better performance guarantees than AOF.

　③. RDB uses snapshots to persist the entire Redis data, while AOF only appends each executed command to the AOF file. Therefore, in theory, RDB is more robust than AOF. According to the official documentation, AOF has some bugs that RDB does not have.

So how should we choose between AOF and RDB persistence methods?

If you can tolerate the loss of data within a short period of time, there is no doubt that using RDB is the best. Generating RDB snapshots regularly is very convenient for database backup, and RDB restores data sets faster than AOF recovery speed is faster, and using RDB can also avoid some hidden bugs of AOF; otherwise, use AOF rewriting. However, it is generally recommended not to use a certain persistence mechanism alone, but to use both together. In this case, when redis restarts, the AOF file will be loaded first to restore the original data, because under normal circumstances The data set saved in the AOF file is more complete than the data set saved in the RDB file. Maybe in the future, Redis officials will merge the two persistence methods into one persistence mode.

The above is the detailed content of Example analysis of AOF persistence in Redis. For more information, please follow other related articles on the PHP Chinese website!