Redis optimization experience summary

Regarding memory management optimization, this article mainly shares with you a summary of Redis optimization experience, hoping to help everyone.

Redis Hash is a HashMap inside the value. If the number of members of the Map is relatively small, a compact format similar to one-dimensional linear will be used to store the Map, which saves a lot of memory overhead of pointers. This Parameter control corresponds to the following 2 items in the redis.conf configuration file:

hash-max-zipmap-entries 64 hash-max-zipmap-value 512                                   When there are 64 members in the value, it will be stored in linear compact format. The default is 64, that is, if there are less than 64 members in the value, linear compact storage will be used. If the value exceeds this value, it will automatically be converted into a real HashMap.

hash-max-zipmap-value means that when the length of each member value in the value Map does not exceed a certain number of bytes, linear compact storage will be used to save space.

If any of the above two conditions exceeds the set value, it will be converted into a real HashMap, which will no longer save memory. So is the bigger the value, the better? The answer is of course no. The advantage of HashMap is that the time complexity of search and operation is O(1). If you give up Hash and use one-dimensional storage, the time complexity is O(n). If the number of

members is small, then The impact is not big, otherwise it will seriously affect the performance, so the setting of this value must be weighed. Generally speaking, it is the most fundamental trade-off between time cost and space cost.

## This list-max-ziplist-value 64 list-max-ziplist-entries 512

List Data type data Type node value is less than how many bytes, which will use compact storage formats, List data types, below A compact storage format without pointers will be used.

Memory pre-allocation:

The internal implementation of Redis has not done much optimization of memory allocation (compared to Memcache). Memory fragmentation will exist to a certain extent, but in most cases this does not It will become the performance bottleneck of Redis. However, if most of the data stored in Redis is numerical, Redis uses a shared integer internally to save the overhead of allocating memory, that is, when the system starts, it first allocates a value from 1 to n Then multiple numerical objects are placed in a pool. If the stored data happens to be within this numerical range, the object is taken directly from the pool and shared through reference counting. This way a large number of numerical values ​​are stored in the system. It can also save memory and improve performance to a certain extent. The setting of this parameter value n requires modifying a line of macro definition REDIS_SHARED_INTEGERS in the source code. The default value is 10000. You can modify it according to your own needs. Just recompile after modification. .

Persistence mechanism:

Timed snapshot method (snapshot):

This persistence method is actually a timer event inside Redis, which checks the current time every fixed time Whether the number and time of data changes meet the configured persistence trigger conditions, if so, a child process will be created through the operating system fork call. This child process will share the same address space with the parent process by default. At this time, you can The entire memory is traversed through the child process to perform storage operations, while the main process can still provide services. When there is a write, the operating system performs copy-on-write in units of memory pages to ensure that there is no connection between the parent and child processes. will affect each other.

The main disadvantage of this persistence is that the scheduled snapshot only represents the memory image within a period of time, so the system restart will lose all data between the last snapshot and the restart.

Statement-based append method (aof):

The aof method is actually similar to mysql's statement-based binlog method, that is, every command that changes Redis memory data will be appended to a log file , which means that this log file is the persistent data of Redis.

The main disadvantage of the aof method is that appending the log file may cause the size to be too large. When the system is restarted to restore data, loading the data will be very slow if the aof method is used. It may take several hours to load dozens of gigabytes of data. After loading, of course, this time-consuming is not because the disk file reading speed is slow, but because all the commands read must be executed in the memory. In addition, since each command requires writing logs, the read and write performance of Redis will also be reduced when using aof.

You can consider saving data to different Redis instances. The memory size of each instance is about 2G. This avoids putting eggs in one basket, which can not only reduce the impact of cache failure on the system, but also It can speed up data recovery, but it also brings a certain complexity to the system design.

Redis persistence crash problem:

People with Redis online operation and maintenance experience will find that Redis will occur when physical memory is used more, but it has not exceeded the actual total physical memory capacity. Some people think that the problem of instability or even crash is caused by the doubling of memory usage due to the fork system call based on snapshot persistence. This view is inaccurate because the copy-on-write mechanism of the fork call is based on the operating system page. In this unit, only dirty pages that have been written will be copied, but generally your system will not have all pages written in a short period of time, resulting in copying. So what causes Redis to crash?

The answer is that Redis persistence uses Buffer IO. The so-called Buffer IO means that Redis uses the Page Cache of physical memory for writing and reading operations on persistent files, while most database systems use Direct IO. To bypass this layer of Page Cache and maintain a data Cache by itself, when the Redis persistence file is too large (especially the snapshot file), and when reading and writing it, the data in the disk file will be loaded into the physical memory. As a layer of cache for the file in the operating system, the data in this layer of cache and the data managed in Redis memory are actually stored repeatedly. Although the kernel will do the work of eliminating the Page Cache when the physical memory is tight, the kernel is likely to If you think a certain Page Cache is more important and let your process start Swap, then your system will begin to become unstable or crash. Our experience is that when your Redis physical memory usage exceeds 3/5 of the total memory capacity, it will become more dangerous.

Summary:

1. Choose the appropriate data type according to business needs, and set corresponding compact storage parameters for different application scenarios.

2. When the business scenario does not require data persistence, turning off all persistence methods can achieve the best performance and maximum memory usage.

3. If you need to use persistence, choose between snapshot mode and statement append mode based on whether you can tolerate the loss of some data during restart. Do not use virtual memory and diskstore mode.

4. Do not let the physical memory usage of the machine where your Redis is located exceed 3/5 of the actual total memory.

The maxmemory option in redis.conf, this option tells Redis to start rejecting subsequent write requests after how much physical memory is used. This parameter can well protect your Redis from using Excessive physical memory will cause swap, which will eventually seriously affect performance or even crash.

vm-enabled in the redis.conf file is no

Related recommendations:

Summary of usage of Redis function in php

Redis cluster construction full record

Summary of common methods for operating redis in PHP

The above is the detailed content of Redis optimization experience summary. For more information, please follow other related articles on the PHP Chinese website!