Detailed explanation of data expiration strategy in Redis

I believe everyone has some understanding of data expiration in Redis. This article mainly introduces the data expiration strategy in Redis. The article introduces it in detail through sample code. I believe it has certain reference value for everyone's understanding and learning. If necessary, Friends can refer to it, I hope it can help everyone.

1. The expiration time of the key in Redis

Use the EXPIRE key seconds command to set the expiration time of the data. Returning 1 indicates that the setting is successful, returning 0 indicates that the key does not exist or the expiration time cannot be set successfully. After setting the expiration time on the key, the key will be automatically deleted after the specified number of seconds. Keys with a specified expiration time are said to be unstable in Redis.

When the key is deleted by the DEL command or reset by the SET or GETSET command, the expiration time associated with it will be cleared

127.0.0.1:6379> setex s 20 1
OK
127.0.0.1:6379> ttl s
(integer) 17
127.0.0.1:6379> setex s 200 1
OK
127.0.0.1:6379> ttl s
(integer) 195
127.0.0.1:6379> setrange s 3 100
(integer) 6
127.0.0.1:6379> ttl s
(integer) 152
127.0.0.1:6379> get s
"1\x00\x00100"
127.0.0.1:6379> ttl s
(integer) 108
127.0.0.1:6379> getset s 200
"1\x00\x00100"
127.0.0.1:6379> get s
"200"
127.0.0.1:6379> ttl s
(integer) -1

Use PERSIST to clear the expiration time

127.0.0.1:6379> setex s 100 test
OK
127.0.0.1:6379> get s
"test"
127.0.0.1:6379> ttl s
(integer) 94
127.0.0.1:6379> type s
string
127.0.0.1:6379> strlen s
(integer) 4
127.0.0.1:6379> persist s
(integer) 1
127.0.0.1:6379> ttl s
(integer) -1
127.0.0.1:6379> get s
"test"

Using rename only changes the key value

127.0.0.1:6379> expire s 200
(integer) 1
127.0.0.1:6379> ttl s
(integer) 198
127.0.0.1:6379> rename s ss
OK
127.0.0.1:6379> ttl ss
(integer) 187
127.0.0.1:6379> type ss
string
127.0.0.1:6379> get ss
"test"

Note: After Redis2.6, the expire precision can be controlled within 0 to 1 millisecond, and the expiration information of the key is stored in the form of absolute Unix timestamp (after Redis2.6, in milliseconds level of precision storage), so when synchronizing multiple servers, the time of each server must be synchronized

2. Redis expired key deletion strategy

There are three ways to expire Redis key:

1. Passive deletion: When reading/writing an expired key, the lazy deletion policy will be triggered and the expired key will be deleted directly

2. Active deletion : Since the lazy deletion strategy cannot guarantee that cold data will be deleted in time, Redis will regularly actively eliminate a batch of expired keys

3. When the current used memory exceeds the maxmemory limit, active cleanup is triggered. Strategy
   

Passive deletion

Only when the key is operated (such as GET), REDIS will passively check whether the key has expired. If it expires, delete it and Return NIL.

1. This deletion strategy is friendly to the CPU. The deletion operation will only be performed when necessary, and unnecessary CPU time will not be wasted on other expired keys.

2. However, this strategy is not friendly to memory. A key has expired, but it will not be deleted before it is operated and still occupies memory space. If a large number of expired keys exist but are rarely accessed, it will cause a lot of waste of memory space. The expireIfNeeded(redisDb *db, robj *key) function is located in src/db.c.

/*-----------------------------------------------------------------------------
 * Expires API
 *----------------------------------------------------------------------------*/
 
int removeExpire(redisDb *db, robj *key) {
 /* An expire may only be removed if there is a corresponding entry in the
 * main dict. Otherwise, the key will never be freed. */
 redisAssertWithInfo(NULL,key,dictFind(db->dict,key->ptr) != NULL);
 return dictDelete(db->expires,key->ptr) == DICT_OK;
}
 
void setExpire(redisDb *db, robj *key, long long when) {
 dictEntry *kde, *de;
 
 /* Reuse the sds from the main dict in the expire dict */
 kde = dictFind(db->dict,key->ptr);
 redisAssertWithInfo(NULL,key,kde != NULL);
 de = dictReplaceRaw(db->expires,dictGetKey(kde));
 dictSetSignedIntegerVal(de,when);
}
 
/* Return the expire time of the specified key, or -1 if no expire
 * is associated with this key (i.e. the key is non volatile) */
long long getExpire(redisDb *db, robj *key) {
 dictEntry *de;
 
 /* No expire? return ASAP */
 if (dictSize(db->expires) == 0 ||
 (de = dictFind(db->expires,key->ptr)) == NULL) return -1;
 
 /* The entry was found in the expire dict, this means it should also
 * be present in the main dict (safety check). */
 redisAssertWithInfo(NULL,key,dictFind(db->dict,key->ptr) != NULL);
 return dictGetSignedIntegerVal(de);
}
 
/* Propagate expires into slaves and the AOF file.
 * When a key expires in the master, a DEL operation for this key is sent
 * to all the slaves and the AOF file if enabled.
 *
 * This way the key expiry is centralized in one place, and since both
 * AOF and the master->slave link guarantee operation ordering, everything
 * will be consistent even if we allow write operations against expiring
 * keys. */
void propagateExpire(redisDb *db, robj *key) {
 robj *argv[2];
 
 argv[0] = shared.del;
 argv[1] = key;
 incrRefCount(argv[0]);
 incrRefCount(argv[1]);
 
 if (server.aof_state != REDIS_AOF_OFF)
 feedAppendOnlyFile(server.delCommand,db->id,argv,2);
 replicationFeedSlaves(server.slaves,db->id,argv,2);
 
 decrRefCount(argv[0]);
 decrRefCount(argv[1]);
}
 
int expireIfNeeded(redisDb *db, robj *key) {
 mstime_t when = getExpire(db,key);
 mstime_t now;
 
 if (when < 0) return 0; /* No expire for this key */ /* Don&#39;t expire anything while loading. It will be done later. */ if (server.loading) return 0; /* If we are in the context of a Lua script, we claim that time is * blocked to when the Lua script started. This way a key can expire * only the first time it is accessed and not in the middle of the * script execution, making propagation to slaves / AOF consistent. * See issue #1525 on Github for more information. */ now = server.lua_caller ? server.lua_time_start : mstime(); /* If we are running in the context of a slave, return ASAP: * the slave key expiration is controlled by the master that will * send us synthesized DEL operations for expired keys. * * Still we try to return the right information to the caller, * that is, 0 if we think the key should be still valid, 1 if * we think the key is expired at this time. */ if (server.masterhost != NULL) return now > when;
 
 /* Return when this key has not expired */
 if (now <= when) return 0; /* Delete the key */ server.stat_expiredkeys++; propagateExpire(db,key); notifyKeyspaceEvent(REDIS_NOTIFY_EXPIRED, "expired",key,db->id);
 return dbDelete(db,key);
}
 
/*-----------------------------------------------------------------------------
 * Expires Commands
 *----------------------------------------------------------------------------*/
 
/* This is the generic command implementation for EXPIRE, PEXPIRE, EXPIREAT
 * and PEXPIREAT. Because the commad second argument may be relative or absolute
 * the "basetime" argument is used to signal what the base time is (either 0
 * for *AT variants of the command, or the current time for relative expires).
 *
 * unit is either UNIT_SECONDS or UNIT_MILLISECONDS, and is only used for
 * the argv[2] parameter. The basetime is always specified in milliseconds. */
void expireGenericCommand(redisClient *c, long long basetime, int unit) {
 robj *key = c->argv[1], *param = c->argv[2];
 long long when; /* unix time in milliseconds when the key will expire. */
 
 if (getLongLongFromObjectOrReply(c, param, &when, NULL) != REDIS_OK)
 return;
 
 if (unit == UNIT_SECONDS) when *= 1000;
 when += basetime;
 
 /* No key, return zero. */
 if (lookupKeyRead(c->db,key) == NULL) {
 addReply(c,shared.czero);
 return;
 }
 
 /* EXPIRE with negative TTL, or EXPIREAT with a timestamp into the past
 * should never be executed as a DEL when load the AOF or in the context
 * of a slave instance.
 *
 * Instead we take the other branch of the IF statement setting an expire
 * (possibly in the past) and wait for an explicit DEL from the master. */
 if (when <= mstime() && !server.loading && !server.masterhost) { robj *aux; redisAssertWithInfo(c,key,dbDelete(c->db,key));
 server.dirty++;
 
 /* Replicate/AOF this as an explicit DEL. */
 aux = createStringObject("DEL",3);
 rewriteClientCommandVector(c,2,aux,key);
 decrRefCount(aux);
 signalModifiedKey(c->db,key);
 notifyKeyspaceEvent(REDIS_NOTIFY_GENERIC,"del",key,c->db->id);
 addReply(c, shared.cone);
 return;
 } else {
 setExpire(c->db,key,when);
 addReply(c,shared.cone);
 signalModifiedKey(c->db,key);
 notifyKeyspaceEvent(REDIS_NOTIFY_GENERIC,"expire",key,c->db->id);
 server.dirty++;
 return;
 }
}
 
void expireCommand(redisClient *c) {
 expireGenericCommand(c,mstime(),UNIT_SECONDS);
}
 
void expireatCommand(redisClient *c) {
 expireGenericCommand(c,0,UNIT_SECONDS);
}
 
void pexpireCommand(redisClient *c) {
 expireGenericCommand(c,mstime(),UNIT_MILLISECONDS);
}
 
void pexpireatCommand(redisClient *c) {
 expireGenericCommand(c,0,UNIT_MILLISECONDS);
}
 
void ttlGenericCommand(redisClient *c, int output_ms) {
 long long expire, ttl = -1;
 
 /* If the key does not exist at all, return -2 */
 if (lookupKeyRead(c->db,c->argv[1]) == NULL) {
 addReplyLongLong(c,-2);
 return;
 }
 /* The key exists. Return -1 if it has no expire, or the actual
 * TTL value otherwise. */
 expire = getExpire(c->db,c->argv[1]);
 if (expire != -1) {
 ttl = expire-mstime();
 if (ttl < 0) ttl = 0; } if (ttl == -1) { addReplyLongLong(c,-1); } else { addReplyLongLong(c,output_ms ? ttl : ((ttl+500)/1000)); } } void ttlCommand(redisClient *c) { ttlGenericCommand(c, 0); } void pttlCommand(redisClient *c) { ttlGenericCommand(c, 1); } void persistCommand(redisClient *c) { dictEntry *de; de = dictFind(c->db->dict,c->argv[1]->ptr);
 if (de == NULL) {
 addReply(c,shared.czero);
 } else {
 if (removeExpire(c->db,c->argv[1])) {
  addReply(c,shared.cone);
  server.dirty++;
 } else {
  addReply(c,shared.czero);
 }
 }
}

But this alone is not enough, because there may be some keys that will never be accessed again. These keys with expiration time set also need to be deleted after expiration. We can even use this The situation is regarded as a memory leak - useless garbage data occupies a large amount of memory, but the server will not release them by itself. This is definitely not a good idea for the Redis server whose running status is very dependent on memory. Message

Active deletion

Let’s talk about time events first. For servers that continue to run, the server needs to regularly check and organize its own resources and status, so that the server can maintain the A healthy and stable state. Such operations are collectively called regular operations (cron job)

In Redis, regular operations are implemented by redis.c/serverCron, which mainly performs the following operations

• Update various statistical information of the server, such as time, memory usage, database usage, etc.

• Clean expired key-value pairs in the database.

• Resize unreasonable databases.

• Close and clean up clients with failed connections.

• Attempt to perform AOF or RDB persistence operation.

• If the server is the master node, perform regular synchronization of the slave nodes.

• If in cluster mode, perform regular synchronization and connection tests on the cluster.

Redis runs serverCron as a time event to ensure that it will run automatically every once in a while, and because serverCron needs to run regularly while the Redis server is running, so it is A cycle time event: serverCron will be executed periodically until the server is shut down.

In the Redis 2.6 version, the program stipulates that serverCron runs 10 times per second, running once every 100 milliseconds on average. Starting from Redis 2.8, users can adjust the number of serverCron executions per second by modifying the hz option. For specific information, please refer to the description of the hz option in the redis.conf file.

is also called scheduled deletion, here is "regular" It refers to the cleanup strategy triggered regularly by Redis, which is completed by the activeExpireCycle(void) function located in src/redis.c.

serverCron is a positioning task driven by the redis event framework. This scheduled task will call the activeExpireCycle function to delete as many expired keys as possible within the limited time REDIS_EXPIRELOOKUPS_TIME_LIMIT for each db. The reason for limiting the time This is to prevent long-term blocking from affecting the normal operation of redis. This active deletion strategy makes up for the memory unfriendliness of the passive deletion strategy.

Therefore, Redis will periodically randomly test a batch of keys with expiration times set and process them. Expired keys tested will be deleted.

The typical method is that Redis does the following steps 10 times per second:

• 随机测试100个设置了过期时间的key

• 删除所有发现的已过期的key

• 若删除的key超过25个则重复步骤1
  

这是一个基于概率的简单算法，基本的假设是抽出的样本能够代表整个key空间，redis持续清理过期的数据直至将要过期的key的百分比降到了25%以下。这也意味着在任何给定的时刻已经过期但仍占据着内存空间的key的量最多为每秒的写操作量除以4.

Redis-3.0.0中的默认值是10，代表每秒钟调用10次后台任务。

除了主动淘汰的频率外，Redis对每次淘汰任务执行的最大时长也有一个限定，这样保证了每次主动淘汰不会过多阻塞应用请求，以下是这个限定计算公式：

#define ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC 25 /* CPU max % for keys collection */ 
... 
timelimit = 1000000*ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/server.hz/100;

hz调大将会提高Redis主动淘汰的频率，如果你的Redis存储中包含很多冷数据占用内存过大的话，可以考虑将这个值调大，但Redis作者建议这个值不要超过100。我们实际线上将这个值调大到100，观察到CPU会增加2%左右，但对冷数据的内存释放速度确实有明显的提高（通过观察keyspace个数和used_memory大小）。

可以看出timelimit和server.hz是一个倒数的关系，也就是说hz配置越大，timelimit就越小。换句话说是每秒钟期望的主动淘汰频率越高，则每次淘汰最长占用时间就越短。这里每秒钟的最长淘汰占用时间是固定的250ms（1000000*ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/100），而淘汰频率和每次淘汰的最长时间是通过hz参数控制的。

从以上的分析看，当redis中的过期key比率没有超过25%之前，提高hz可以明显提高扫描key的最小个数。假设hz为10，则一秒内最少扫描200个key（一秒调用10次*每次最少随机取出20个key），如果hz改为100，则一秒内最少扫描2000个key；另一方面，如果过期key比率超过25%，则扫描key的个数无上限，但是cpu时间每秒钟最多占用250ms。

当REDIS运行在主从模式时，只有主结点才会执行上述这两种过期删除策略，然后把删除操作”del key”同步到从结点。

maxmemory

当前已用内存超过maxmemory限定时，触发主动清理策略

• volatile-lru：只对设置了过期时间的key进行LRU（默认值）

• allkeys-lru ： 删除lru算法的key

• volatile-random：随机删除即将过期key

• allkeys-random：随机删除

• volatile-ttl ： 删除即将过期的

• noeviction ： 永不过期，返回错误当mem_used内存已经超过maxmemory的设定，对于所有的读写请求，都会触发redis.c/freeMemoryIfNeeded(void)函数以清理超出的内存。注意这个清理过程是阻塞的，直到清理出足够的内存空间。所以如果在达到maxmemory并且调用方还在不断写入的情况下，可能会反复触发主动清理策略，导致请求会有一定的延迟。

当mem_used内存已经超过maxmemory的设定，对于所有的读写请求，都会触发redis.c/freeMemoryIfNeeded(void)函数以清理超出的内存。注意这个清理过程是阻塞的，直到清理出足够的内存空间。所以如果在达到maxmemory并且调用方还在不断写入的情况下，可能会反复触发主动清理策略，导致请求会有一定的延迟。

清理时会根据用户配置的maxmemory-policy来做适当的清理（一般是LRU或TTL），这里的LRU或TTL策略并不是针对redis的所有key，而是以配置文件中的maxmemory-samples个key作为样本池进行抽样清理。

maxmemory-samples在redis-3.0.0中的默认配置为5，如果增加，会提高LRU或TTL的精准度，redis作者测试的结果是当这个配置为10时已经非常接近全量LRU的精准度了，并且增加maxmemory-samples会导致在主动清理时消耗更多的CPU时间，建议：

• 尽量不要触发maxmemory，最好在mem_used内存占用达到maxmemory的一定比例后，需要考虑调大hz以加快淘汰，或者进行集群扩容。

• 如果能够控制住内存，则可以不用修改maxmemory-samples配置；如果Redis本身就作为LRU cache服务（这种服务一般长时间处于maxmemory状态，由Redis自动做LRU淘汰），可以适当调大maxmemory-samples。
  

以下是上文中提到的配置参数的说明

# Redis calls an internal function to perform many background tasks, like 
# closing connections of clients in timeout, purging expired keys that are 
# never requested, and so forth. 
# 
# Not all tasks are performed with the same frequency, but Redis checks for 
# tasks to perform according to the specified "hz" value. 
# 
# By default "hz" is set to 10. Raising the value will use more CPU when 
# Redis is idle, but at the same time will make Redis more responsive when 
# there are many keys expiring at the same time, and timeouts may be 
# handled with more precision. 
# 
# The range is between 1 and 500, however a value over 100 is usually not 
# a good idea. Most users should use the default of 10 and raise this up to 
# 100 only in environments where very low latency is required. 
hz 10 
 
# MAXMEMORY POLICY: how Redis will select what to remove when maxmemory 
# is reached. You can select among five behaviors: 
# 
# volatile-lru -> remove the key with an expire set using an LRU algorithm 
# allkeys-lru -> remove any key according to the LRU algorithm 
# volatile-random -> remove a random key with an expire set 
# allkeys-random -> remove a random key, any key 
# volatile-ttl -> remove the key with the nearest expire time (minor TTL) 
# noeviction -> don't expire at all, just return an error on write operations 
# 
# Note: with any of the above policies, Redis will return an error on write 
# operations, when there are no suitable keys for eviction. 
# 
# At the date of writing these commands are: set setnx setex append 
# incr decr rpush lpush rpushx lpushx linsert lset rpoplpush sadd 
# sinter sinterstore sunion sunionstore sdiff sdiffstore zadd zincrby 
# zunionstore zinterstore hset hsetnx hmset hincrby incrby decrby 
# getset mset msetnx exec sort 
# 
# The default is: 
# 
maxmemory-policy noeviction 
 
# LRU and minimal TTL algorithms are not precise algorithms but approximated 
# algorithms (in order to save memory), so you can tune it for speed or 
# accuracy. For default Redis will check five keys and pick the one that was 
# used less recently, you can change the sample size using the following 
# configuration directive. 
# 
# The default of 5 produces good enough results. 10 Approximates very closely 
# true LRU but costs a bit more CPU. 3 is very fast but not very accurate. 
# 
maxmemory-samples 5

Replication link和AOF文件中的过期处理

In order to obtain correct behavior without causing consistency problems, DEL operations when a key expires will be recorded in the AOF file and passed to all related slaves. That is to say, the expired deletion operation is performed uniformly in the master instance and passed down, instead of being controlled by each salve individually. This way there will be no data inconsistencies. When the slave is connected to the master, it cannot immediately clean up the expired keys (it needs to wait for the DEL operation passed by the master). The slave still needs to manage and maintain the expired status in the data set so that when the slave is promoted to master, it can behave like the master. Expiration processing is also performed independently.

Related recommendations:

Summary of usage of Redis function in php

Redis cluster construction full record

Summary of common methods for operating redis in php

The above is the detailed content of Detailed explanation of data expiration strategy in Redis. For more information, please follow other related articles on the PHP Chinese website!