Detailed introduction to the concepts, principles and usage scenarios of message queues (with cases)
- angryTomforward
- 2019-11-23 17:02:263514browse
As we all know, when designing a website, you will encounter "group text messages" to users, "a large number of logs in the order system", "flash sale design", etc. The server cannot handle these. This kind of instantaneous burst of pressure requires the help of "message queue" to ensure the normal and effective use of the system. This article mainly studies through the idea of message queue.
Mainly understand the following knowledge:
1. What is a queue and what can it do?
2. What are the application scenarios for alignment?
3. How to use queues to uncouple services?
4. How to use Redis queue to eliminate high pressure?
5. How to use the professional alignment system RabbitMQ?
The main contents are summarized as follows
@The concept, principle and scenario of message queue
@Decoupling case: queue processing order system and distribution system
@Traffic Peak Shaving Case: Redis’s List Type Realizes Flash Sales
@RabbitMQ: A More Professional Message System Implementation Solution
1. Understanding the Message Queue
1.1 Message queue concept
Essentially, message queue is a middleware with a queue structure, which means that the message can be returned directly after being put into this middleware. The system does not need to process it immediately, but another program will read the data and process it one by one in order.
That is to say, when you encounter a situation that is extremely concurrent and takes a long time, and you do not need to return the processing results immediately, using message queues can solve such problems.
1.2 Core structure
Enqueues messages by a business system, inserts messages into the message queue one by one, and directly returns a successful result after the insertion is successful. There will be a message processing system in the future, which will take out and process the records in the message system one by one, completing a dequeuing process.
1.3 Application Scenario
Data redundancy: For example, in the order system, strict data conversion and recording are required in the future. The message queue can store these data persistently in the queue, and then there are orders , the subsequent processing program obtains it, and after the subsequent processing is completed, the record is deleted to ensure that each record can be processed.
System decoupling: After using the message system, the enqueuing system and the dequeuing system are separated, which means that as long as it crashes one day, it will not affect the normal operation of the other system.
Traffic peak shaving: For example, flash sales and rush sales, we can use message queues in conjunction with caching, which can effectively withstand the amount of instantaneous visits and prevent the server from being overwhelmed and causing a crash.
Asynchronous communication: The message itself can be returned directly after being queued.
Scalability: For example, the order queue can not only process orders, but can also be used by other businesses.
Sorting Guarantee: Some scenarios need to be processed in the order of products, such as single in and single out to ensure that data is processed in a certain order. It is possible to use message queues.
The above are common usage scenarios of message queue. Of course, message queue is just a middleware and can be used in conjunction with other products.
1.4 Common queue implementation advantages and disadvantages
Queue medium
1. Database, such as mysql (high reliability, easy to implement, slow speed)
2 , cache, such as redis (fast, low efficiency when a single message package is too large)
3, message system, such as rabbitMq (highly professional, reliable, high learning cost)
Message Processing trigger mechanism
1. Infinite loop reading: easy to implement, unable to recover in time in case of failure; (more suitable for flash sale, more centralized, centralized operation and maintenance)
2. Scheduled tasks : Pressure is evenly distributed, with a processing upper limit; currently a popular processing trigger mechanism. (The only disadvantage is that you need to pay attention to the interval and data. Don't wait until the previous task is not completed and the next task starts again)
3. Daemon process: similar to php-fpm and php-cg, requires shell basics
2. Decoupling Case: Queue Processing "Order System" and "Distribution System"
For the order process, We can design two systems, one is the "order system" and the other is the "delivery system". We should all have seen it when shopping online. After I submit an order, I can see in the background that my goods are being delivered. middle. At this time, a "delivery system" needs to be involved.
If we design the "order system" and "delivery system" together when doing the architecture, there will be some problems. First of all, for the order system, the pressure on the system will be relatively high, but " The distribution system" does not necessarily have to respond immediately to these pressures.
Secondly, we don’t want the delivery system to malfunction after the order system malfunctions, which will affect the normal operation of both systems at the same time. So we hope to decouple these two systems. After the two systems are separated, we can communicate between the two systems through an intermediate "queue table".
2.1 Architecture design
1. First, the order system will receive the user's order, and then process the order.
2. These order information will then be written to the queue table. This queue table is the key to communicating between the two systems.
3. A program executed regularly by the distribution system reads the queue table for processing.
4. After processing by the distribution system, the processed records will be marked.
2.2 Program flow
#3. Traffic peak clipping case: Redis’s list type realizes flash sales
redis Based on memory, its speed will be very fast. Redis is a very good supplement to the database because it is durable. Redis will periodically write data to the hard disk, so it does not have to worry about power outages. In this respect, it has more advantages than another cache memcache. In addition, redis provides five data types (string, doubly linked list, hash, set, ordered set)
In general, do flash sales Redis is a good choice for cases where cases, rush purchases, and cases that require queuing are instantly higher than yours.
3.1 The list type in the redis data type
The list in redis is a doubly linked list, and data can be appended from the head or tail.
* LPUSH/LPUSHX: Insert the value into the head of the (/existing) list
* RPUSH/RPUSHX: Insert the value into the tail of the (/existing) list
* LPOP: Remove and get the first element of the list
* RPOP: Remove and get the last element of the list
* LTRIM: Keep the elements in the specified range
* LLEN: Get the length of the list
* LSET: Set the value of the list element by index
* LINDEX: Get the element in the list by index
* LRANGE: Get the elements within the specified range of the list
3.2 Architecture design
A simple structure flash kill program design.
#1. First record which user participated in the flash sale and record his time.
2. Save the user's ID in the redis list and let it queue up. If it is stipulated that only the first 10 users can successfully participate, if the number in the list is enough, it will not be allowed to continue to add data. In this way, the length of the redis list will only be 10
3. Finally, slowly write the data in redis into the database to reduce the pressure on the data
3.3 Code-level design
1. When the user starts the flash sale, write the request of the flash sale program into Redis (uid, time_stamp).
2. If it is stipulated that only 10 people can succeed in the flash sale, check the length of the data stored in Redis. If it exceeds the upper limit and discard it directly, it means the flash sale is completed.
3. Finally, the 10 pieces of data stored in Redis are processed in an infinite loop, and then the data is slowly fetched and stored in the mysql database.
In the flash sale area, the pressure on the database is particularly high. If we do not have such a design, it will cause a writing bottleneck in MySQL. We use a queue list in Redis, and then put the flash sale request into Redis. Finally, we slowly write the data into the database through the warehousing program. In this way, the traffic can be balanced and there will be no impact on mysql. Too much pressure.
4. RabbitMQ
Here we will explain some uses of RabbitMQ. First of all, when we talked about the flash sale case before, we mentioned the lock mechanism to prevent other programs from processing the same record. If our system architecture is very complex, there are multiple programs reading a queue in real time, or I have multiple sending programs that operate one or more queues at the same time, and I even want these programs to be distributed on different machines. In this case, using redis queue is somewhat inadequate. What to do at this time? We need to introduce some more professional message queue systems, which can better solve the problem.
4.1 The architecture and principles of RabbitMQ
Work queue
This not only solves the decoupling between producers and consumers, but also enables the sharing of consumers and tasks, reducing the pressure on the server.
5. Summary
The above mainly focuses on learning the concepts, principles, and scenarios of message queues. Decoupling cases and understanding the simple use of RabbitMQ.
6. Question
What is the biggest difference between redis and message server selection.
My understanding is that Redis processes requests one by one. Redis is a single thread. It is different from the message server IO implementation. One is synchronous and the other is asynchronous, while redis uses synchronous blocking, while the message server Use asynchronous non-blocking.
For more Redis related knowledge, please visit the Redis usage tutorial column!
The above is the detailed content of Detailed introduction to the concepts, principles and usage scenarios of message queues (with cases). For more information, please follow other related articles on the PHP Chinese website!