Redis optimization example analysis

Memory Dimension

Control the length of key

The key is generally a string, and the underlying data structure of the string is SDS. The SDS structure will include the string length, allocation Metadata information such as space size. When the length of the key string increases, the metadata in SDS will also occupy more memory space. In order to reduce the space occupied by the key, we can use the corresponding English abbreviation according to the business name to represent it. For example, user is represented by u, and message is represented by m.

Avoid storing bigkey

We need to pay attention to both the length of the key and the size of the value. Redis uses a single thread to read and write data. The read and write operations of bigkey will block the thread and reduce the cost. Redis processing efficiency.

How to query bigkey

We can use the --bigkey command to view the bigkey information occupied in Redis. The specific command is as follows:

redis-cli -h 127.0.0.1 -p 6379 -a 'xxx' --bigkeys

As shown in the above figure, we can see that the key in Redis occupies 32098 bytes and needs to be optimized.

Recommendation:

• If the key is of string type, it is recommended that the size of the value stored in the value is about 10KB.

• If the key is of List/Hash/Set/ZSet type, it is recommended that the number of stored elements be controlled below 10,000.

Choose the appropriate data type

Redis is optimized for the data type it stores and also optimizes the memory accordingly. For relevant knowledge about data results, you can refer to previous articles.

For example: String and set will use integer encoding when storing int data. Hash and ZSet will use compressed list (ziplist) storage when the number of elements is relatively small, and will be converted into hash tables and jump tables when a relatively large amount of data is stored.

Adopt efficient serialization and compression methods

Strings in Redis are saved using binary-safe byte arrays, so we can serialize the business into binary and write it to Redis , but using different serialization, the space occupied is different. Protostuff's serialization is more efficient than Java's built-in serialization and takes up less space. In order to reduce space usage, we can compress and store JSON and XML data formats. Optional compression algorithms include Gzip and Snappy.

Set Redis maximum memory and elimination strategy

We estimate the memory size in advance based on the amount of business data, so as to avoid the continuous expansion of Redis memory, resulting in excessive resource usage.

Regarding how to set the elimination strategy, you need to select based on actual business characteristics:

• ##volatile-lru / allkeys-lru: Prioritize the most recently accessed data

• ##volatile-lfu / allkeys-lfu:

  Prioritize the most frequently accessed data

• volatile-ttl:

  Prioritize the elimination of data that is about to expire

• volatile-random / allkeys-random:

  Random elimination Data

Control the size of the Redis instance

It is recommended that the memory size of a single Redis instance be set between 2 and 6GB. Because RDB snapshots and master-slave cluster data synchronization can be completed quickly, the processing of normal requests will not be blocked.

Clear memory fragments regularly

Frequent new modifications will lead to an increase in memory fragments, so memory fragments need to be cleaned up in time.

Redis provides the Info memory command to view memory usage information, as follows:

Instructions:

used_memory_rss is the physical memory space actually allocated to Redis by the operating system.
used_memory is the space actually applied for by Redis to save data.
mem_fragmentation_ratio=used_memory_rss/ used_memory
mem_fragmentation_ratio is greater than 1 but less than 1.5. This situation is reasonable.
If mem_fragmentation_ratio is greater than 1.5, it means that the memory fragmentation rate has reached more than 50%. In this case, it is usually necessary to take some measures to reduce the memory fragmentation rate. Specific memory cleaning measures will be explained in subsequent articles.
Performance Dimension

It is prohibited to use KEYS, FLUSHALL, and FLUSHDB commands

KEYS matches according to the key content and returns matching matches The key-value pair of the condition. This command requires a full table scan of the Redis global hash table, which seriously blocks the Redis main thread.
FLUSHALL deletes all data on the Redis instance. If the amount of data is large, it will seriously block the Redis main thread.
FLUSHDB, deletes the data in the current database. If the amount of data is large, it will block the Redis main thread.
Optimization suggestions

We need to disable these commands online. The specific method is that the administrator uses the rename-command command to rename these commands in the configuration file so that the client cannot use these commands.

慎用全量操作的命令

对于集合类型的来说，在未清楚集合数据大小的情况下，慎用查询集合中的全量数据，例如Hash的HetALL、Set的SMEMBERS命令、LRANGE key 0 -1 或者ZRANGE key 0 -1等命令，因为这些命令会对Hash或者Set类型的底层数据进行全量扫描，当集合数据量比较大时，会阻塞Redis的主线程。

优化建议：

当元素数据量较多时，可以用SSCAN、HSCAN 命令分批返回集合中的数据，减少对主线程的阻塞。

慎用复杂度过高命令

Redis执行复杂度过高的命令，会消耗更多的 CPU 资源，导致主线程中的其它请求只能等待。常见的复杂命令如下：SORT、SINTER、SINTERSTORE、ZUNIONSTORE、ZINTERSTORE 等聚合类命令。

优化建议：

当需要执行排序、交集、并集操作时，可以在客户端完成，避免让Redis进行过多计算，从而影响Redis性能。

设置合适的过期时间

Redis通常用于保存热数据。热数据一般都有使用的时效性。因此，在数据存储的过程中，应根据业务对数据的使用时间合理地设置数据的过期时间。否则写入Redis的数据会一直占用内存，如果数据持续增增长，会达到机器的内存上限，造成内存溢出，导致服务崩溃。

采用批量命令代替个命令

当我们需要一次性操作多个key时，可以使用批量命令来处理，批量命令可以减少客户端与服务端的来回网络IO次数。

• String或者Hash类型可以使用 MGET/MSET替代 GET/SET，HMGET/HMSET替代HGET/HSET

• 其它数据类型使用Pipeline命令，一次性打包发送多个命令到服务端执行。

Pipeline具体使用：

redisTemplate.executePipelined(new RedisCallback<String>() {
            @Override
            public String doInRedis(RedisConnection connection) throws DataAccessException {
                for (int i = 0; i < 5; i++) 
                {
                    connection.set(("test:" + i).getBytes(), "test".getBytes());
                }
                return null;
            }
        });

高可用维度

按照业务部署不同的实例

不同的业务线来部署 Redis 实例，这样当其中一个实例发生故障时，不会影响到其它业务。

避免单点问题

业务上根据实际情况采用主从、哨兵、集群方案，避免单点故障，影响业务的正常使用。

合理的设置相关参数

针对主从环境，我们需要合理设置相关参数，具体内容如下：

• 合理的设置repl-backlog参数：如果repl-backlog设置过小，当写流量比较大的场景下，主从复制中断可能会引发全量复制数据的风险。

• 合理设置slave client-output-buffer-limit：当从库复制发生问题时，过小的 buffer会导致从库缓冲区溢出，从而导致复制中断。

The above is the detailed content of Redis optimization example analysis. For more information, please follow other related articles on the PHP Chinese website!