A brief analysis of RDB and AOF persistence, what are the advantages and disadvantages? How to choose?

This article will talk to you about the principles of RDB and AOF persistence in redis. What are their advantages and disadvantages? Analyze which one should be used? I hope to be helpful!

Redis provides two persistence solutions: RDB and AOF:

• RDB: Generates Redis in-memory data snapshot is a binary file dumpr.rdb

• AOF: records all Redis write commands except queries, and re-executes these commands when the Redis service starts. Restore data.

RDB persistence

By default, Redis will persist data over a period of time to the hard disk in the form of RDB snapshots and save them as adumpr.rdb Binary file. [Related recommendations: Redis video tutorial]

A brief introduction to the working principle:

When Redis needs to be persisted, Redis will fork a Child process, the child process writes data to a temporary RDB file on the disk. When the child process finishes writing the temporary file, replace the original RDB. The advantage of this is that it can copy-on-write.

Of course we can also manually execute save or bgsave (asynchronously) to generate RDB files.

redis.conf default configuration

save 900  1
save 300  10
save 60  10000

• If more than 1 key is modified within 900 seconds, snapshot saving will be initiated;
• Within 300 seconds, If more than 10 keys are modified, a snapshot save is initiated;
• Within 60 seconds, if 10,000 keys are modified, a snapshot save is initiated;

RDB snapshot command

By default, Redis saves the database snapshot in a binary file named dump.rdb.

You can set Redis to automatically save a data set when the condition "the data set has at least M changes within N seconds" is met.

You can also manually let Redis save the data set by calling SAVE or BGSAVE.

For example, the following settings will allow Redis to automatically save a data set when it meets the condition of "at least 1000 keys have been changed within 60 seconds":

save 60 1000

This kind of persistence This method is called a snapshot.

RDB Creation Principle

When Redis needs to save the dump.rdb file, the server performs the following operations:

• Redis calls fork() and has both parent and child processes.
• The subprocess writes the data set to a temporary RDB file.
• When the child process completes writing to the new RDB file, Redis replaces the original RDB file with the new RDB file and deletes the old RDB file.

This way of working allows Redis to benefit from the copy-on-write mechanism.

Advantages of RDB

RDB is a relatively compact file that saves Redis data at a certain point in time. This kind of data More suitable for backup and disaster recovery.

For example, you can back up an RDB file every hour within the last 24 hours, and also back up an RDB file every day of the month. This way, even if you encounter a problem, you can always restore the data set to a different version.

Disadvantages of RDB

If you need to minimize data loss in the event of a server failure, then RDB is not for you. Although Redis allows you to set different save points to control the frequency of saving RDB files, it is not an easy operation because RDB files need to save the state of the entire data set. So you may save the RDB file at least once every 5 minutes. In this case, in the event of an outage, you may lose several minutes of data.

AOF persistence

Use AOF for persistence. Each write command is appended to appendonly.aof# through the write function. ## file.

AOF can achieve full persistence. It only needs to be turned on in the configuration file (the default is no).

appendfsync yes After turning on AOF, every time Redis executes a command to modify data, it will Add it to the AOF file, and when Redis restarts, the AOF file will be read and "replayed" to restore the last moment before Redis was shut down.

AOF configuration

You can configure how often Redis fsyncs data to disk.

redis.conf default configuration

appendfsync yes
appendfsync always     #每次有数据修改发生时都会写入AOF文件。
appendfsync everysec   #每秒钟同步一次，该策略为AOF的缺省策略。

has three options:

1, execute fsync every time a new command is appended to the AOF file : very slow and very safe.
2, fsync once per second: fast enough (almost the same as using RDB persistence), and only 1 second of data will be lost in the event of a failure.
3, never fsync: Leave the data to the operating system for processing. Faster and less secure option.

The recommended (and default) measure is to fsync once per second. This fsync strategy can take into account both speed and security.

AOF creation principle

AOF rewriting is the same as RDB snapshot creation, both of which cleverly utilize the copy-on-write mechanism.

The following are the execution steps of AOF rewriting:

Redis executes fork() and now has both the parent process and the child process.

The child process starts writing the contents of the new AOF file to the temporary file.

For all newly executed write commands, the parent process accumulates them into a memory cache and appends these changes to the end of the existing AOF file: This way, even if a shutdown occurs in the middle of a rewrite, Existing AOF files are still safe.

When the child process completes the rewriting work, it sends a signal to the parent process. After receiving the signal, the parent process appends all the data in the memory cache to the end of the new AOF file.

Done! Redis now atomically replaces the old file with the new one, and all commands thereafter are appended directly to the end of the new AOF file.

Advantages of AOF

1. Using AOF for persistence, you can set different fsync strategies, such as no fsync, once per second fsync, or fsync every time a write command is executed.

The default policy of AOF is to fsync once per second. Under this configuration, Redis can still maintain good performance, and even if a failure occurs, only one second of data will be lost at most.

fsync will be executed on a background thread, so the main thread can continue to work hard processing command requests.

2. The AOF file is a log file that only performs append operations. It is not one that is replaced after generating a new one. Even if the log contains incomplete commands for some reasons (such as when writing The disk is full, writing is stopped midway, etc.), the redis-check-aof tool can also easily fix this problem.

3, Redis can automatically rewrite the AOF in the background when the AOF file size becomes too large: the new AOF file after rewriting contains the minimum set of commands required to restore the current data set.

The entire rewriting operation is absolutely safe, because Redis rewriting creates a new AOF file. During the rewriting process, commands will continue to be appended to the existing old AOF file, even if an error occurs during the rewriting process. Even if there is a shutdown, the existing old AOF files will not be lost. Once the new AOF file is created, Redis will switch from the old AOF file to the new AOF file and start appending to the new AOF file.

4. The AOF file stores all write operations performed on the database in an orderly manner. These write operations are saved in the format of the Redis protocol, so the content of the AOF file is very easy to read. Analysis (parse) is also easy. Exporting AOF files is also very simple: for example, if you accidentally execute the _FLUSH ALL_ (clear the data of the entire Redis server (delete all keys in all databases).) command, but as long as the AOF file has not been overwritten , then as long as you stop the server, remove the FLUSHALL command at the end of the AOF file, and restart Redis, you can restore the data set to the state before FLUSHALL was executed.

Disadvantages of AOF

For the same data set, the size of the AOF file is usually larger than the size of the RDB file.

Depending on the fsync strategy used, AOF may be slower than RDB. Under normal circumstances, fsync performance per second is still very high, and turning off fsync can make AOF as fast as RDB, even under heavy load.

However, when dealing with huge write loads, RDB can provide a more guaranteed maximum latency (latency).

The difference between RDB and AOF

RDB persistence refers to writing a snapshot of the data set in memory to disk within a specified time interval. The actual operation process It is to fork a child process and first write the data set into a temporary file. After the writing is successful, the previous file is replaced and stored using binary compression.

AOF persistence records every write and delete operation processed by the server in the form of a log. Query operations will not be recorded. Records are appended in the form of text. You can open the file to see detailed operation records.

RDB or AOF Which one should I use?

If you care deeply about your data but can still afford data loss within minutes, then you can just use RDB persistence.

AOF appends every command executed by Redis to the disk. Processing huge writes will reduce the performance of Redis. I don’t know if you can accept it.

Database backup and disaster recovery:

Regularly generating RDB snapshots is very convenient for database backup, and RDB restores data sets faster than AOF.

Redis supports opening RDB and AOF at the same time. After the system restarts, Redis will give priority to using AOF to restore data, so that the data lost will be the least.

AOF BGREWRITEAOF rewrite

Because AOF works by continuously appending commands to the end of the file, so as the number of written commands continues to increase, the AOF file The size will also become larger and larger.

For example

If you call INCR 100 times on a counter, then just to save the current value of the counter, the AOF file needs Use 100 records (entry).

However, in reality, using only one SET command is enough to save the current value of the counter, and the remaining 99 records are actually redundant.

In order to handle this situation, Redis supports an interesting feature: the AOF file can be reconstructed without interrupting the service client.

Execute the BG REWRITE AOF command, Redis will generate a new AOF file, which contains the minimum commands required to reconstruct the current data set.

Redis 2.2 needs to manually execute the BGREWRITEAOF command; Redis 2.4 can automatically trigger AOF rewriting. For specific information, please see the sample configuration file of 2.4.

Backup Redis data

Disk failure, node failure, and other problems may cause your data to disappear. Failure to back up is very dangerous.

Redis is very friendly for data backup, because you can copy the RDB file while the server is running: Once the RDB file is created, no modifications will be made. When the server wants to create a new RDB file, it first saves the contents of the file in a temporary file. When the temporary file is written, the program uses rename(2) to atomically replace the original RDB file with the temporary file.

This means that it is absolutely safe to copy RDB files at any time.

The following are our suggestions:

1. Create a regular task (cron job) to back up an RDB file to a folder every hour, and backup it every day. Back up an RDB file to another folder.

2. Make sure that the snapshot backups have corresponding date and time information. Every time you execute the regular task script, use the find command to delete expired snapshots: For example, you can keep the snapshots within the last 48 hours. Hourly snapshots, and daily snapshots for the last one or two months can also be retained.

3. At least once a day, back up the RDB outside your data center, or at least outside the physical machine where you are running the Redis server.

For more programming related knowledge, please visit: Programming Video! !

The above is the detailed content of A brief analysis of RDB and AOF persistence, what are the advantages and disadvantages? How to choose?. For more information, please follow other related articles on the PHP Chinese website!