Let me give you an example. We have built a mysql binlog
synchronization system before. The pressure is still very high. The daily synchronization data has to reach hundreds of millions, which means that the data is intact from a mysql database. Synchronize to another mysql library (mysql -> mysql). A common point is that, for example, a big data team needs to synchronize a mysql library to perform various complex operations on the data of the company's business system.
When you add, delete or modify a piece of data in mysql, 3 logs of the addition, deletion or modification will appear binlog
, and then these three binlog
will be sent to MQ, and then consumed in sequence When executing, you must at least ensure that people come in order, right? Otherwise, it was originally: add, modify, delete; but you just changed the order and executed it to delete, modify, and add. Isn't that completely wrong?
Originally, when this data was synchronized, the data should have been deleted at the end; but as a result, you got the order wrong, and the data was retained in the end, and the data synchronization went wrong.
Let’s take a look at the two scenarios where the order will be out of order:
RabbitMQ: one queue, multiple consumers. For example, the producer sends three pieces of data to RabbitMQ, in the order of data1/data2/data3, and what is pushed into a memory queue of RabbitMQ. There are three consumers who consume one of the three pieces of data from MQ. As a result, consumer 2 completes the operation first and saves data2 to the database, followed by data1/data3. This isn't obviously messed up.
Kafka: For example, we created a topic with three partitions. When the producer writes, he can actually specify a key. For example, if we specify an order id as the key, then the data related to this order will definitely be distributed to the same partition, and the data in this partition must be There is an order.
When consumers take out data from the partition, they must also be in order. At this point, the order is still ok and there is no confusion. Next, we may create multiple threads in the consumer to process messages concurrently. Because if the consumer consumes and processes in a single thread, and the processing is time-consuming, for example, it takes tens of ms to process a message, then only dozens of messages can be processed in 1 second, which is too low throughput. If multiple threads run concurrently, the order may be messed up.
solution
RabbitMQ
Split multiple queues, each queue has one consumer, which is just more queues, which is indeed a troublesome point; or there is only one queue but corresponding to one consumer, and then this consumer uses a memory queue internally for queuing, and then distributes Give the bottom layer different workers to handle.
Kafka
One topic, one partition, one consumer, internal single-thread consumption, single-thread throughput is too low, generally this will not be used.
Write N memory queues, and data with the same key will go to the same memory queue; then for N threads, each thread consumes one memory queue respectively, so that Ensure orderliness.
The above is the detailed content of How does mysql ensure the order of messages?. For more information, please follow other related articles on the PHP Chinese website!

本篇文章给大家带来了关于mysql的相关知识,其中主要介绍了关于架构原理的相关内容,MySQL Server架构自顶向下大致可以分网络连接层、服务层、存储引擎层和系统文件层,下面一起来看一下,希望对大家有帮助。

在mysql中,可以利用char()和REPLACE()函数来替换换行符;REPLACE()函数可以用新字符串替换列中的换行符,而换行符可使用“char(13)”来表示,语法为“replace(字段名,char(13),'新字符串') ”。

方法:1、利用right函数,语法为“update 表名 set 指定字段 = right(指定字段, length(指定字段)-1)...”;2、利用substring函数,语法为“select substring(指定字段,2)..”。

mysql的msi与zip版本的区别:1、zip包含的安装程序是一种主动安装,而msi包含的是被installer所用的安装文件以提交请求的方式安装;2、zip是一种数据压缩和文档存储的文件格式,msi是微软格式的安装包。

转换方法:1、利用cast函数,语法“select * from 表名 order by cast(字段名 as SIGNED)”;2、利用“select * from 表名 order by CONVERT(字段名,SIGNED)”语句。

本篇文章给大家带来了关于mysql的相关知识,其中主要介绍了关于MySQL复制技术的相关问题,包括了异步复制、半同步复制等等内容,下面一起来看一下,希望对大家有帮助。

本篇文章给大家带来了关于mysql的相关知识,其中主要介绍了mysql高级篇的一些问题,包括了索引是什么、索引底层实现等等问题,下面一起来看一下,希望对大家有帮助。

在mysql中,可以利用REGEXP运算符判断数据是否是数字类型,语法为“String REGEXP '[^0-9.]'”;该运算符是正则表达式的缩写,若数据字符中含有数字时,返回的结果是true,反之返回的结果是false。


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

EditPlus Chinese cracked version
Small size, syntax highlighting, does not support code prompt function

Safe Exam Browser
Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

MantisBT
Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

SublimeText3 English version
Recommended: Win version, supports code prompts!

SublimeText3 Mac version
God-level code editing software (SublimeText3)
