Data recovery from server crash using redis

Because redis is stored in memory and provides data structure storage types commonly used in general programming languages, it is often used for data recovery processing when servers crash.

The server can store the data that needs to be saved in redis in the form of json objects in certain specified processes, which is what we often call snapshots. When the server is running, redis is read to determine whether it needs to be restored. The data continues to be processed.

Just delete the redis data after a business process is completed.

Redis provides two methods for exporting memory data to the hard disk for data backup:

RDB method (default)

The persistence of RDB method is through snapshotting (snapshotting) Completed, when certain conditions are met, Redis will automatically snapshot all data in the memory and store it on the hard disk. The conditions for taking a snapshot can be customized by the user in the configuration file and consist of two parameters: time and the number of changed keys. A snapshot will be taken when the number of keys changed within the specified time is greater than the specified value. RDB is the default persistence method used by redis. Three conditions have been preset in the configuration file:

save 900 1 # If at least 1 key is changed within 900 seconds, a snapshot will be taken

save 300 10 # Take a snapshot if at least 10 keys have been changed within 300 seconds

save 60 10000 # Take a snapshot if at least 10,000 keys have been changed within 60 seconds

There can be multiple There are several conditions, and there is an "OR" relationship between the conditions. As long as one of the conditions is met, a snapshot will be taken. If you want to disable automatic snapshots, just delete all save parameters.

Redis will store the snapshot file in the dump.rdb file in the current directory (which can be viewed by CONFIG GET dir) by default. You can specify the storage path and file name of the snapshot file by configuring the dir and dbfilename parameters respectively. .

The process of Redis implementing snapshots

Redis uses the fork function to copy a copy (child process) of the current process (parent process);

The parent process continues to receive and process customers The command is sent from the terminal, and the child process starts to write the data in the memory to the temporary file on the hard disk;

When the child process finishes writing all the data, the old RDB file will be replaced with the temporary file, so far A snapshot operation is completed.

When executing fork, the operating system (Unix-like operating system) will use the copy-on-write strategy. That is, the parent and child processes share the same memory data at the moment the fork function occurs. When the parent process wants to When a certain piece of data is changed (such as executing a write command), the operating system will copy the piece of data to ensure that the data of the child process is not affected, so the new RDB file stores the memory data at the moment the fork is executed.

Redis will not modify the RDB file during the snapshot process. It will replace the old file with a new one only after the snapshot is completed, which means that the RDB file is complete at any time. This allows us to implement Redis database backup by regularly backing up RDB files. The RDB file is a compressed binary format (the rdbcompression parameter can be configured to disable compression to save CPU usage), so the space occupied will be smaller than the data size in the memory, making it easier to transmit.

In addition to automatic snapshots, you can also manually send the SAVE or BGSAVE command to let Redis perform the snapshot. The difference between the two commands is that the former is performed by the main process and will block other requests, while the latter will be performed through fork The child process performs snapshot operations. After Redis is started, it will read the RDB snapshot file and load the data from the hard disk to the memory. This time varies depending on the size and structure of the data and server performance. It usually takes 20 to 30 seconds to load a 1GB snapshot file that records 10 million string type keys into memory. Persistence is achieved through RDB. Once Redis exits abnormally, all data changed after the last snapshot will be lost. This requires developers to control possible data loss within an acceptable range by combining and setting automatic snapshot conditions based on specific application scenarios. If the data is so important that it cannot afford any loss, you can consider using the AOF method for persistence.

AOF method

By default, Redis does not enable AOF (append only file) persistence. It can be enabled through the appendonly parameter in redis.conf:

appendonly yes

When starting, Redis will execute the commands in the AOF file one by one to load the data in the hard disk into the memory. The loading speed will be slower than RDB.

Enable AOF persistence Every time a command is executed that changes the data in Redis, Redis will write the command to the AOF file on the hard disk. The saving location of the AOF file is the same as the location of the RDB file, both are set through the dir parameter. The default file name is appendonly.aof, which can be modified through the appendfilename parameter:

appendfilename appendonly.aof

Configure the conditions for redis to automatically rewrite AOF files

auto-aof-rewrite-percentage 100 # When the current AOF file size exceeds the percentage of the AOF file size during the last rewrite, it will be performed again Rewrite, if it has not been rewritten before, it will be based on the AOF file size at startup

auto-aof-rewrite-min-size 64mb #The minimum AOF file size allowed to be rewritten

Configure the mechanism that requires the system to refresh the hard disk cache after writing the AOF file

# appendfsync always # Synchronization will be performed every time a write is performed, which is the safest and slowest

appendfsync everysec # Perform a synchronization operation every second

# appendfsync no # Does not actively perform synchronization operations, but leaves it entirely to the operating system (i.e. once every 30 seconds), which is the fastest and least secure

Redis allows AOF and RDB to be opened at the same time, which not only ensures data security but also makes backup and other operations very easy. After restarting Redis at this time, Redis will use the AOF file to restore the data, because AOF persistence may lose less data

redis = require('redis'),//导入js模块
RDS_PORT = ,        //端口号
RDS_HOST = '',    //服务器IP
RDS_OPTS = {},            //设置项
redisdb = redis.createClient(RDS_PORT, RDS_HOST, RDS_OPTS);//创建连接
 
redisdb.select(20);//指定分区库
 
redisdb.on('ready', function (res) {
    console.log('ready');
});
 
redisdb.on('connect', function () {
    console.log('connect');
});
exports.redisdb = redisdb;
 
function redis_opt(opt, key, value, callback) {
 
    if (opt == 'get') {
        redisdb.get(key, function (err, data) {
            if (err == null) {
                callback(data);
            }
            else {
                callback(err);
            }
        });
    }
    else if (opt == 'set')
    {
        redisdb.set(key,value, function (err,result) {
            if (err == null) {
                callback(result);
            }
            else {
                callback(err);
            }
        });
    }
    else if (opt == 'del')
    {
        redisdb.del(key, function (err, result) {
            if (err == null) {
                callback(result);
            }
            else {
                callback(err);
            }
        });
    }
    else
    {
        callback("error opt!");
    }
    
}
 
function update(key)
{
    redis_opt("get", key, null, function (data) {
        console.log("the redis data is " + data);
        if (data) {
            count = parseInt(data);
            redis_opt("set", key, ++count , function (data) {
                console.log("set " + count  + " " + data);
            });  
        }
        else {
            redis_opt("set", key, 10000, function (data) {
                console.log("set " + 10000 + " " + data);
            });  
        }
    });
 
 
 
 
 
    
 
 
   
 
 
}
 
function clear(key)
{
    redis_opt("del", key, null, function (ret) {
        console.log("del " + key + " " + ret);
    });  
}
 
 
 
function main()
{
 
    var key = "count_test";
 
    setInterval(function () { clear(key) }, 5000);
 
    setInterval(function () { update(key) }, 1000);
 
}
 
 
 
 
//testmain();
 
 
main();

The above code is a simple timer function, that is, it is read regularly after the server is started. The redis data is cumulatively modified if it exists, and initialized if it does not exist. At the same time, for the convenience of explanation, a timer is set to delete data regularly.

For more redis knowledge, please pay attention to the redis introductory tutorial column.

The above is the detailed content of Data recovery from server crash using redis. For more information, please follow other related articles on the PHP Chinese website!