How to delete data in redis?

Due to changes in requirements, a previous project required modifications to the data format stored in redis. To prevent old data from being inserted into new data after the new package is released. Therefore, all old data must be deleted before publishing. Currently redis is a public cluster involving several businesses. Then the question arises, how to delete a large amount of old data (the total number of keys in the library is currently 12 million) without affecting the use of other businesses.

Common ways to delete redis data in batches:

If the key of the data to be deleted is known, you can Use the del command of redis-cli /usr/local/redis/bin/redis-cli del key or you can also use the redis package or library corresponding to other high-level languages. For example, jedis under java and redis library under python

java:   jdeis.del(key)
python: redis.delete(key)

If the key of the data to be deleted is unknown, only the key that satisfies a specific pattern is known. In this case, you need to use the keys command of redis to find the keys that satisfy a specific pattern

Find all keys that meet the prefix of video

/usr/local/redis/bin/redis-cli keys video_*

You can use Linux's xargs to complete batch deletion/ usr/local/redis/bin/redis-cli keys video* | xargs /usr/local/redis/bin/redis-cli del 3. If the data to be deleted is all the data in the library, you can use flushdb to clear the entire library/ usr/local/redis/bin/redis-cli flushdb

Instructions for several methods

The first method requires clear knowledge of the specific key

Use the keys command. When the amount of data in the library is too large, the keys command will block all other requests for redis. Undoubtedly, this approach is not advisable for public redis clusters. Of course, specific business needs must be considered. If that doesn't work, you can also execute the deletion script at a time when the business traffic is relatively small.

Using flushdb will clean the data in the entire library.

My solution The online redis cluster uses the matser-slave structure. Therefore, the keys command that blocks the request can be executed on the slave node to find all keys that meet the specific prefix. Then use a shell script or high-level language to delete the data on the master node.

#Get all the keys whose prefix is ​​video, album, actor, and append these keys to the file /data/keys.txt

#!/bin/bash

keys=('video' 'album' 'actor');
host='localhost';
port='6378';
for key in ${keys[@]};
do
  cmd="/usr/local/redis/bin/redis-cli -h ${host} -p ${port} keys gal.video.${key}*  >> /data/keys.txt";
  echo ${cmd}; 
  eval ${cmd};
done;
# 根据前面生成的key，删除数据
#!/bin/bash
host='localhost';
port='6378';
file="/data/keys.txt";
i=0;
cat ${file} | while read key;
do
  let i=i+1;
  cmd="/usr/local/redis/bin/redis-cli -h ${host} -p ${port} del ${key}";
  echo "line:"${i}",cmd:"${cmd};
  eval ${cmd}; 
done;

Since script 2 sends del commands one by one, the execution efficiency is quite low. In the test, about 1.2 million pieces of data were deleted in one hour. 12 million items need to be deleted in 10 hours! ! ! Considering the time it takes to send each request, I thought of using the redis pipeline to implement batch submission. ###

__author__ = 'litao'
from redis import Redis
host="127.0.0.1"
port=6379
db=0
r =Redis(host,port,db)
pl=r.pipeline()
per_pipe_size=10000
count=0
file = open("/data/keys.txt")
print "start del all keys in "+file.name
while 1:
    lines = file.readlines(10000)
    if not lines:
        break
    for key in lines:
        key=key.strip('\n')
        pl.delete(key)
        count=count+1
        if(count==per_pipe_size):
            count=0
            pl.execute()
pl.execute()
file.close()
print 'finish del all keys'

###The improved script 2 only takes about 2 minutes to execute online! ! ###

The above is the detailed content of How to delete data in redis?. For more information, please follow other related articles on the PHP Chinese website!