How to solve the problem that Redis bigkeys command will block

1. An SF Express senior development engineer executed a dangerous Redis command online, causing a company to lose 4 million.

One command resulted in a loss of millions. Does this require compensation?

The code is not standardized and my colleagues burst into tears. Please be careful when coding!

Out of curiosity, let me test it out. What is the problem?

2. Test the performance of 10 million data

1. Write a script file

Write 10 million data.

for((i=1;i<=10000000;i++)); do echo "set k$i 哪吒编程$i" >> /tmp/test1.txt;done;

Check whether the writing is successful through /tmp/test1.txt.

2. Write 10 million data to Redis

cat /tmp/test1.txt | redis-cli -a 111111 --pipe

3. View 10 million data through keys *

4. Prohibit the use of keys * through the configuration file

Configure security in the redis.conf file:

  rename- command keys ""
  rename- command flushdb ""
  rename- command flushall ""

3. Use scan instead of keys *

The Redis Scan command is used to iterate over database keys in the database.

The SCAN command is a cursor-based iterator. Each time it is called, a new cursor will be returned to the user. The user needs to use this new cursor as the cursor parameter of the SCAN command in the next iteration. In this way to continue the previous iterative process.

SCAN returns an array containing two elements. The first element is the new cursor for the next iteration, and the second element is an array containing all the iterated items. element. If the new cursor returns 0 it indicates that the iteration has ended.

scan syntax:

SCAN cursor [MATCH pattern] [COUNT count]

4. Reject bigkey

1. Alibaba Cloud Redis development specifications

Alibaba Cloud The Redis development specifications clearly stipulate"Reject bigkey (prevent network card traffic and slow queries)".

String type is controlled within 10KB, and the number of hash, list, set, and zset elements should not exceed 5000.

2. How to delete bigkey when it appears?

1. String type is deleted with del.

2. Other types use hscan, sscan, and zscan methods to progressively delete. At the same time, the problem of automatic deletion of bigkey expiration time must be avoided, because it will cause the main thread to block.

Hash Delete: hscan hdel

public void delBigHash(String host, int port, String password, String bigHashKey) {
    Jedis jedis = new Jedis(host, port);
    if (password != null && !"".equals(password)) {
        jedis.auth(password);
    }
    ScanParams scanParams = new ScanParams().count(100);
    String cursor = "0";
    do {
        ScanResult<Entry<String, String>> scanResult = jedis.hscan(bigHashKey, cursor, scanParams);
        List<Entry<String, String>> entryList = scanResult.getResult();
        if (entryList != null && !entryList.isEmpty()) {
            for (Entry<String, String> entry : entryList) {
                jedis.hdel(bigHashKey, entry.getKey());
            }
        }
        cursor = scanResult.getStringCursor();
    } while (!"0".equals(cursor));
    
    //删除 bigkey
    jedis.del(bigHashKey);
}

3. What problems will bigkey cause?

1. Memory is uneven, cluster migration is difficult;

2. Timeout deletion, blocking threads;

3. Network traffic is blocked;

4. How to find bigkey?

(1) Search through redis-cli --bigkeys.

(2) Calculate the number of bytes of each key value and search through the memory usage key

The above is the detailed content of How to solve the problem that Redis bigkeys command will block. For more information, please follow other related articles on the PHP Chinese website!