search
HomeDatabaseRedisLet's analyze the solutions to Redis hot data problems together

This article brings you relevant knowledge about Redis, which mainly introduces the related issues about the Redis hot key large Value solution. Let’s take a look at it together. I hope it will be helpful to everyone. .

Let's analyze the solutions to Redis hot data problems together

Recommended learning: Redis video tutorial

About Redis hot data & big key big value questions are also easy to ask high-level questions It is better to learn the problem at once and leave the interviewer speechless. In my personal work experience, hot data problems are more likely to be encountered at work than avalanches. However, most of the time, the hot spots are not hot enough and will be alerted and solved in advance. But once this problem cannot be controlled, the online problems caused will be enough to put your performance at the bottom this year. Let’s not talk nonsense and get to the point.

Under normal circumstances, data in the Redis cluster is evenly distributed to each node, and requests are evenly distributed to each shard. However, in some special scenarios, such as external crawlers, attacks, and hot products Wait, the most typical example is when a celebrity announces their divorce on Weibo, and people flood in to leave messages, causing the Weibo comment function to crash. In this short period of time, the number of visits to certain keys is too large, and requests will be made for the same keys. to the same data shard, resulting in a high load on the shard and becoming a bottleneck, leading to a series of problems such as avalanches.

1. Interviewer: Have you ever encountered Redis hot data problems in your project? What are the common causes?

Problem analysis: I was asked this question last time when I heard a big boss in the group interview Ali p7. The difficulty index is five stars. Wait for me. Being a noob is really a plus.

Answer: I have something to say about hot data. I have been aware of this problem since I first learned to use Redis, so when using it We will deliberately avoid and resolutely not dig holes for ourselves. The biggest problem with hotspot data will cause failures caused by unbalanced load in the Redis cluster (that is, data skew). These problems are fatal blows to the Redis cluster.

Let’s first talk about the main reasons for the Reids cluster load imbalance failure:

  1. Highly visited Key, that is, hot key, according to past maintenance If the QPS accessed by a key exceeds 1,000, you should pay close attention to it, such as popular products, hot topics, etc.
  2. Big Value, although the access QPS of some keys is not high, due to the large value, the network card load is large, the network card traffic is full, and a single machine may experience Gigabit/second , IO failure.
  3. Hot Key Big Value exists at the same time, server killer.

So what faults will be caused by hot keys or large Values:

  1. Data skew problem: Large Values ​​will cause uneven data distribution on different nodes in the cluster, causing data skew problems , a large number of requests with a very high read-write ratio will fall on the same redis server, and the load of the redis will be seriously increased, making it easy to crash.
  2. QPS Skew: QPS is uneven across shards.
  3. A large Value will cause the Redis server buffer to be insufficient, resulting in get timeout.
  4. Because the Value is too large, the computer room network card has insufficient traffic.
  5. Redis cache failure leads to a chain reaction of database layer breakdown.

2. Interviewer: In real projects, how do you accurately locate hot data issues?

Answer: The solution to this problem is relatively broad. It depends on different business scenarios. For example, if a company organizes promotional activities, then there must be a way to count the products participating in the promotion in advance. , this scenario can pass the estimation method. For emergencies and uncertainties, Redis will monitor hotspot data by itself. To sum up:

  1. How to know in advance:
    Depending on the business, human flesh statistics or system statistics may become hot data, such as promotional products, hot topics, Holiday topics, anniversary activities, etc.
  2. Redis client collection method:
    The caller counts the number of key requests by counting, but the number of keys cannot be predicted, and the code is highly intrusive.
    public Connection sendCommand(final ProtocolCommand cmd, final byte[]... args) {
        //从参数中获取key
        String key = analysis(args);
        //计数
        counterKey(key);
        //ignore
    }
  3. Redis cluster proxy layer statistics:
    Agent-based Redis distributed architectures such as Twemproxy and codis have a unified entrance and can be collected and reported at the Proxy layer. But the disadvantage is obvious. Not all Redis cluster architectures have proxies.

    Lets analyze the solutions to Redis hot data problems together

  4. Redis server collection:
    Monitor the QPS of a single Redis shard, and monitor the node where the QPS is tilted to a certain extent to obtain the hotspot key. Redis provides a monitor command, you can count all the commands on a certain Redis node within a period of time and analyze the hot key. Under high concurrency conditions, there will be hidden dangers of memory explosion and Redis performance, so this method is suitable for use in a short period of time; it is also only It can count the hotspot keys of a Redis node. For clusters, summary statistics are needed, which is a little troublesome from a business perspective.

    Lets analyze the solutions to Redis hot data problems together

    The above four methods are commonly used in the industry. I have a new idea by studying the Redis source code. Type 5: Modify the Redis source code.

  5. Modify the Redis source code: (Thinking of ideas from reading the source code)

    I found that Redis4.0 has brought us many new things Features, including the LFU-based hotspot key discovery mechanism. With this new feature, we can implement hotspot key statistics on this basis. This is just my personal idea.

    Interviewer's psychology: The young man is quite thoughtful and broad-minded, and he also pays attention to modifying the source code. I don't have this ambition. We need people like this in our team.

    (Discover problems, analyze problems, solve problems, and directly tell how to solve hot data problems without waiting for the interviewer to ask questions. This is the core content)

3. How to solve hot data issues

Answer: Regarding how to manage hot data issues, we mainly consider two aspects to solve this problem. The first is data sharding, so that The pressure is evenly distributed to multiple shards of the cluster to prevent a single machine from hanging. The second is migration isolation.

Summary summary:

  1. key split:
    If the current key type is a secondary data structure, such as Hash type. If the number of hash elements is large, you can consider splitting the current hash so that the hot key can be split into several new keys and distributed to different Redis nodes, thereby reducing the pressure
  2. Migrate hotspot key:
    Taking Redis Cluster as an example, you can migrate the slot where the hotspot key is located to a new Redis node separately. In this way, even if the QPS of this hotspot key is very high, it will not affect the entire cluster. Other businesses can also be customized and developed, and hotspot keys are automatically migrated to independent nodes. This solution is also more multi-copy.
  3. Hotspot key current limit:
    For read commands, we can solve the problem by migrating the hotspot key and then adding slave nodes. For write commands, we can limit the current by targeting this hotspot key separately. .
  4. Increase local cache:
    For businesses whose data consistency is not that high, you can cache the hotspot key in the local cache of the business machine, because it is in the local memory of the business end. A remote IO call is eliminated. However, when the data is updated, it may cause inconsistency between the business and Redis data.

Interviewer: You answered very well and considered it very comprehensively.

4. Interviewer: Regarding the last question about Redis, Redis supports rich data types, so how to solve the problem of large Value stored in these data types? Have you encountered this situation online?

Problem analysis: Compared with the big concept of hot key, the concept of big Value is better to understand. Since Redis runs in a single thread, if the value of an operation is very large, it will affect the entire operation. The response time of redis has a negative impact because Redis is a Key-Value structure database. A large value means that a single value takes up a large amount of memory. The most direct impact on the Redis cluster is data skew.

Answer: (You want to stump me? I am prepared.)

Let me first talk about how big the Value is, which is given based on the company's infrastructure. The experience value can be divided as follows:

Note: (The experience value is not a standard, it is summarized based on long-term observation of online cases by cluster operation and maintenance personnel)

  1. Big: string type value > 10K, number of elements in set, list, hash, zset and other collection data types > 1000.
  2. Extra large: string type value > 100K, the number of elements in set, list, hash, zset and other collection data types > 10000.

Since Redis runs in a single thread, if the value of an operation is very large, it will have a negative impact on the response time of the entire redis. Therefore, it can be split if it can be split in terms of business. Here are a few typical splits: Demolition plan:

  1.  一个较大的 key-value 拆分成几个 key-value ,将操作压力平摊到多个 redis 实例中,降低对单个 redis 的 IO 影响
  2. 将分拆后的几个 key-value 存储在一个 hash 中,每个 field 代表一个具体的属性,使用 hget,hmget 来获取部分的 value,使用 hset,hmset 来更新部分属性。
  3. hash、set、zset、list 中存储过多的元素

类似于场景一中的第一个做法,可以将这些元素分拆。

以 hash 为例,原先的正常存取流程是:

hget(hashKey, field); 
hset(hashKey, field, value)

现在,固定一个桶的数量,比如 10000,每次存取的时候,先在本地计算 field 的 hash 值,模除 10000,确定该 field 落在哪个 key 上,核心思想就是将 value 打散,每次只 get 你需要的。

newHashKey = hashKey + (hash(field) % 10000); 
hset(newHashKey, field, value); 
hget(newHashKey, field)

推荐学习:Redis学习教程

The above is the detailed content of Let's analyze the solutions to Redis hot data problems together. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:CSDN. If there is any infringement, please contact admin@php.cn delete
Redis's Role: Exploring the Data Storage and Management CapabilitiesRedis's Role: Exploring the Data Storage and Management CapabilitiesApr 22, 2025 am 12:10 AM

Redis plays a key role in data storage and management, and has become the core of modern applications through its multiple data structures and persistence mechanisms. 1) Redis supports data structures such as strings, lists, collections, ordered collections and hash tables, and is suitable for cache and complex business logic. 2) Through two persistence methods, RDB and AOF, Redis ensures reliable storage and rapid recovery of data.

Redis: Understanding NoSQL ConceptsRedis: Understanding NoSQL ConceptsApr 21, 2025 am 12:04 AM

Redis is a NoSQL database suitable for efficient storage and access of large-scale data. 1.Redis is an open source memory data structure storage system that supports multiple data structures. 2. It provides extremely fast read and write speeds, suitable for caching, session management, etc. 3.Redis supports persistence and ensures data security through RDB and AOF. 4. Usage examples include basic key-value pair operations and advanced collection deduplication functions. 5. Common errors include connection problems, data type mismatch and memory overflow, so you need to pay attention to debugging. 6. Performance optimization suggestions include selecting the appropriate data structure and setting up memory elimination strategies.

Redis: Real-World Use Cases and ExamplesRedis: Real-World Use Cases and ExamplesApr 20, 2025 am 12:06 AM

The applications of Redis in the real world include: 1. As a cache system, accelerate database query, 2. To store the session data of web applications, 3. To implement real-time rankings, 4. To simplify message delivery as a message queue. Redis's versatility and high performance make it shine in these scenarios.

Redis: Exploring Its Features and FunctionalityRedis: Exploring Its Features and FunctionalityApr 19, 2025 am 12:04 AM

Redis stands out because of its high speed, versatility and rich data structure. 1) Redis supports data structures such as strings, lists, collections, hashs and ordered collections. 2) It stores data through memory and supports RDB and AOF persistence. 3) Starting from Redis 6.0, multi-threaded I/O operations have been introduced, which has improved performance in high concurrency scenarios.

Is Redis a SQL or NoSQL Database? The Answer ExplainedIs Redis a SQL or NoSQL Database? The Answer ExplainedApr 18, 2025 am 12:11 AM

RedisisclassifiedasaNoSQLdatabasebecauseitusesakey-valuedatamodelinsteadofthetraditionalrelationaldatabasemodel.Itoffersspeedandflexibility,makingitidealforreal-timeapplicationsandcaching,butitmaynotbesuitableforscenariosrequiringstrictdataintegrityo

Redis: Improving Application Performance and ScalabilityRedis: Improving Application Performance and ScalabilityApr 17, 2025 am 12:16 AM

Redis improves application performance and scalability by caching data, implementing distributed locking and data persistence. 1) Cache data: Use Redis to cache frequently accessed data to improve data access speed. 2) Distributed lock: Use Redis to implement distributed locks to ensure the security of operation in a distributed environment. 3) Data persistence: Ensure data security through RDB and AOF mechanisms to prevent data loss.

Redis: Exploring Its Data Model and StructureRedis: Exploring Its Data Model and StructureApr 16, 2025 am 12:09 AM

Redis's data model and structure include five main types: 1. String: used to store text or binary data, and supports atomic operations. 2. List: Ordered elements collection, suitable for queues and stacks. 3. Set: Unordered unique elements set, supporting set operation. 4. Ordered Set (SortedSet): A unique set of elements with scores, suitable for rankings. 5. Hash table (Hash): a collection of key-value pairs, suitable for storing objects.

Redis: Classifying Its Database ApproachRedis: Classifying Its Database ApproachApr 15, 2025 am 12:06 AM

Redis's database methods include in-memory databases and key-value storage. 1) Redis stores data in memory, and reads and writes fast. 2) It uses key-value pairs to store data, supports complex data structures such as lists, collections, hash tables and ordered collections, suitable for caches and NoSQL databases.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Atom editor mac version download

Atom editor mac version download

The most popular open source editor

SublimeText3 Linux new version

SublimeText3 Linux new version

SublimeText3 Linux latest version

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.