Redis command usage example analysis

Cause of the problem

The application the editor is responsible for is a management background application. Shiro framework is used for permission management. Since there are multiple nodes, distributed Session needs to be used, so Redis is used to store Session information.

Since Shiro does not directly provide the Redis storage Session component, Afan had to use shiro-redis, an open source component from Github.

Since the Shiro framework needs to regularly verify whether the Session is valid, the bottom layer of Shiro will call SessionDAO#getActiveSessions to obtain all Session information.

And shiro-redis happens to inherit the SessionDAO interface. The bottom layer uses the keys command to find all the Session## stored in Redis. #key.

public Set<byte[]> keys(byte[] pattern){
    checkAndInit();
    Set<byte[]> keys = null;
    Jedis jedis = jedisPool.getResource();
    try{
        keys = jedis.keys(pattern);
    }finally{
        jedis.close();
    }
    return keys;
}

If you find the cause of the problem, the solution is relatively simple. Find the solution on github and upgrade

shiro-redis to the latest version.

In this version,

shiro-redis uses the scan command instead of keys to fix this problem.

public Set<byte[]> keys(byte[] pattern) {
    Set<byte[]> keys = null;
    Jedis jedis = jedisPool.getResource();


    try{
        keys = new HashSet<byte[]>();
        ScanParams params = new ScanParams();
        params.count(count);
        params.match(pattern);
        byte[] cursor = ScanParams.SCAN_POINTER_START_BINARY;
        ScanResult<byte[]> scanResult;
        do{
            scanResult = jedis.scan(cursor,params);
            keys.addAll(scanResult.getResult());
            cursor = scanResult.getCursorAsBytes();
        }while(scanResult.getStringCursor().compareTo(ScanParams.SCAN_POINTER_START) > 0);
    }finally{
        jedis.close();
    }
    return keys;


}

Although the problem was successfully solved, Ah Fen was still a little confused.

Why does the

keys command cause other commands to execute slower?

Why

Keys command query is so slow?

Why is there no problem with the

Scan command?

The principle of Redis command execution

First of all, let’s look at the first question, why does the

keys instruction cause the execution of other commands to slow down?

From the perspective of the client, executing a command is divided into three steps:

1. Send the command

2. Execute the command

3. Return results

But this is only the process that the client thinks, but in fact, at the same time, there may be many clients sending commands to Redis. We all know that Redis uses a single-threaded model.

In order to process all client request commands at the same time, Redis uses a queue internally for queue execution.

So the client actually needs four steps to execute a command:

1. Send the command

2. Command queuing

3. Execute command

4. Return results

Since Redis executes commands in a single thread, it can only start by taking out tasks from the queue sequentially implement.

As long as 3 this process executes the command too slowly, other tasks in the queue have to wait. To the external client, Redis seems to be blocked and never responds.

So when using the Redis process, do not execute instructions that require long running. This may cause Redis to block and affect the execution of other instructions.

KEYS Principle

Let’s start to answer the second question, why is the

Keys command query so slow?

Before answering this question, please recall the underlying storage structure of Redis.

It doesn’t matter if you don’t know your friend’s. You can look back at the previous article “Alibaba Interviewer: Are you familiar with HashMap? Okay, let’s talk about the Redis dictionary!”

keys The command needs to return all Redis keys that match the given pattern pattern. In order to achieve this purpose, Redis has to traverse the dictionary ht[ 0]The underlying array of the hash table, this time complexity is 『O(N)』 (N is the number of keys in Redis).

Even if the number of keys in Redis is small, it will still have fast execution speed. When the number of Redis keys gradually increases and reaches millions, tens of millions, or even hundreds of millions, its execution speed will become very slow.

The following is an experiment done by Ah Fen locally. Use lua script to add 10W keys to Redis, and then use

keys to query all keys. This query will block for about ten seconds. .

eval "for i=1,100000  do redis.call('set',i,i+1) end" 0

Here Afan uses Docker to deploy Redis, and the performance may be slightly worse.

SCAN Principle

Finally let’s look at the third question, why is there no problem with the

scan command?

This is because the

scan command uses a black technology-"cursor-based iterator".

Every time the

scan command is called, Redis will return a new cursor and a certain number of keys to the user. If you want to continue to obtain the remaining keys next time, you need to pass this cursor to the scan command to continue the previous iteration process.

To put it simply, the

scan command uses paging to query redis.

The following is an example of the iterative process of the scan command:

scan The command uses cursors to cleverly split a full query into multiple times, reducing the complexity of the query. Spend.

Although the time complexity of the

scan command is the same as keys, both are 「O(N)」, but due to scan The command only needs to return a small number of keys, so the execution speed will be very fast.

Finally, although the

scan command solves the keys shortcomings, it also introduces some other defects:

• The same element It may be returned multiple times, which requires our application to add the function of handling repeated elements.

• During the iteration process, elements being added to Redis or elements being deleted may be returned, or they may not be returned.

The above defects need to be considered in our development.

In addition to scan, redis has several other commands for incremental iteration:

• sscan: use Used to iterate the database keys in the current database, used to solve the smemberspossible blocking problem

• hscan command is used to iterate the hash keys Key-value pairs are used to solve hgetall possible blocking problems.

• zscan: The command is used to iterate the elements in the ordered set (including element members and element scores), and is used to generate zrange Blocking problems may occur.

The above is the detailed content of Redis command usage example analysis. For more information, please follow other related articles on the PHP Chinese website!