search
HomeDatabaseMysql TutorialChat MySQL memory management, memory allocator, and operating systems
Chat MySQL memory management, memory allocator, and operating systemsJan 08, 2021 am 09:35 AM
mysqlmemory allocatorMemory managementoperating system

Chat MySQL memory management, memory allocator, and operating systems

Recommended (free): mysql video tutorial

When users encounter memory problems using any software (including MySQL), Our first reaction is a memory leak. As this article shows, that’s not always the case.

This article explains a bug about memory.

All Percona Support customers are eligible for bug fixes, but their options vary. For example, Advanced customers are offered a HotFix build prior to the public release of software with the patch. Premium customers do not even have to use Percona software: we may port our patches to upstream for them. But for Percona products all Support levels have the right to have a fix.

all percona supported customers All are eligible for bug fixes, but they also have different options. For example, VIP customers can get the hotfiix version before the software patch is officially released. Premium customers don't even need to use Percona's software. We can also push the patch upstream for them. But for Percona products, all support levels are entitled to bug fixes.

Even so, this does not mean we will fix every unexpected behavior, even if we accept that behavior to be a valid bug. One of the reasons for such a decision might be that while the behavior is clearly wrong for Percona products, this is still a feature request.

Even so, this does not mean that we will fix all unexpected situations, even if we accept this situation as a valid bug. One of the reasons for making such a decision may be that although this unexpected situation is clearly wrong, it is indeed a product requirement for the percona product itself

as a bug

A good recent example of such a case is PS-5312 – the bug is repeatable with upstream and reported at bugs.mysql.com/95065

A good recent example of such a case is PS-5312— —This bug can be reproduced upstream and documented at bugs.mysql.com/95065.

This reports a situation whereby access to InnoDB fulltext indexes leads to growth in memory usage. It starts when someone queries a fulltext index, grows until a maximum, and is not freed for quite a long time.

This report describes a situation that causes memory usage to increase when accessing InnoDB's full-text index. This situation occurs in some full-text index queries. The memory will continue to grow until it reaches the maximum value and will not be released for a long time.

Yura Sorokin from the Percona Engineering Team investigated if this is a memory leak and found that it is not.

Yura Sorokin from the Percona Engineering Team investigated if this is a memory leak and found that it is not. Leakage scope.

When InnoDB resolves a fulltext query, it creates a memory heap in the function fts_query_phrase_search This heap may grow up to 80MB. Additionally, it has a big number of blocks ( mem_block_t ) which are not always used continuously and this , in turn, leads to memory fragmentation.

When InnoDB parses a full-text query, it creates a memory heap in the fts_query_phrase_search function, which may grow to 80M. In addition, this process will also use a large number of non-contiguous blocks (mem_block_t), resulting in memory fragmentation.

In the function exit , the memory heap is freed. InnoDB does this for each of the allocated blocks. At the end of the function, it calls free() which belongs to one of the memory allocator libraries, such as malloc or jemalloc. From the mysqld point of view, everything is done correctly: there is no memory leak.

At function exit, these memory heaps will be released. InnoDB does this for every block it allocates. At the end of function execution, call a free() operation in a memory allocator library, such as malloc or jemalloc. From the perspective of MySQL itself, this is no problem and there is no memory leak.

However while free() should release memory when called, it is not required to return it back to the operating system. If the memory allocator decides that the same memory blocks will be required soon, it may still keep them for the mysqld process. This explains why you might see that mysqld still uses a lot of memory after the job is finished and all de-allocations are done.

However, the free() function should indeed be freed when it is called memory, but does not need to return it to the operating system. If the memory allocator finds that these memory blocks are needed immediately, they will be reserved for the mysqld process. This explains why mysqld still takes up a lot of memory after completing its work and releasing memory.

This in practice is not a big issue and should not cause any harm. But if you need the memory to be returned to the operating system quicker, you could try alternative memory allocators, such as jemalloc. The latter was proven to solve the issue with PS-5312.

This is not a big problem in actual production, and it should not cause any accidents. But if you need to return memory to the operating system faster, you can try a non-traditional memory allocator, like jemallolc. It is proven to solve the problem of PS-5312.

Another factor which improves memory management is the number of CPU cores: the more we used for the test, the faster the memory was returned to the operating system. This, probably, can be explained by the fact that if you have multiple CPUs, then the memory allocator can dedicate one of them just for releasing memory to the operating system.

Another factor that improves memory management is the number of cpu cores: in testing, the more cpu cores , the memory will be returned to the operating system faster. It's possible that you have multiple CPUs and one of them can be used exclusively as a memory allocator to free up memory for the operating system.

The very first implementation of InnoDB full text indexes introduced this flaw. As our engineer Yura Sorokin found:

  • ##The very first 5.6 commit which introduces Full Text Search Functionality for InnoDB WL#5538: InnoDB Full-Text Search Support – https://dev.mysql.com/worklog/task/?id=5538

  • Implement WL #5538 InnoDB Full-Text Search Support, merge – https://github.com/mysql/mysql-server/commit/b6169e2d944 – also has this problem.

As our engineers

Yura Sorokin is the same as what was found. The following two points illustrate that the early implementation of InnoDB full-text index introduced this flaw:

  • 5.6 version of MySQL was the first to implement InnoDB WL full-text index Introduction to the function introduction: #5538: InnoDB full-text search support – https://dev.mysql.com/worklog/task/?id=5538

  • Implementing WL #5538 InnoDB full-text search Support and Merger - https://github.com/mysql/mysql-server/commit/b6169e2d944 - The same problem also exists Problem

Fixing method

We have a few options to fix this:

  1. Change implementation of InnoDB fulltext index

  2. Use custom memory library like jemalloc

Both have their advantages and disadvantages.

We have two ways to fix this problem:

1. Modify the implementation of InnoDB full-text index

2. Use a custom memory library such as

jemalloc

Both methods have their own pros and cons.

Option 1 means we are introducing an incompatibility with upstream, which may lead to strange bugs in future versions. This also means a full rewrite of the InnoDB fulltext code which is always risky in GA versions, used by our customers .

Method 1 means we introduce the risk of incompatibility with the software upstream, which may lead to unknown bugs in new versions. It also means completely rewriting part of the InnoDB full-text index code, which is risky in the GA version used by users.

Option 2 means we may hit flaws in the jemalloc library which is designed for performance and not for the safest memory allocation.

Option 2 means we may hit some A bug in the jemalloc library that is designed for performance but is not the safest memory allocation.

So we have to choose between these two not ideal solutions.

Since option 1 may lead to a situation when Percona Server will be incompatible with upstream, we prefer option 2and look forward for the upstream fix of this bug.

So we have to choose one of these two imperfect methods.

In view of the fact that method one may cause the percona service to be incompatible with the upstream, we prefer to use method two to solve the problem and look forward to the upstream fixing this bug.

Conclusion

If you are seeing a high memory usage by the mysqld process, it is not always a symptom of a memory leak. You can use memory instrumentation in Performance Schema to find out how allocated memory is used. Try alternative memory libraries for better processing of allocations and freeing of memory. Search the user manual for LD_PRELOAD to find out how to set it up at these pages here and here.

If you find that the mysqld process takes up a lot of memory, it does not necessarily mean that there is a memory leak. We can use memory instrumentation in Performance Schema to understand how the process uses allocated memory. You can also try replacing the memory library to better handle memory allocation and deallocation. For how to configure LD_RELOAD, please refer to the corresponding pages of the MySQL user manual mysqld-safe and using-system.

For more programming-related knowledge, please visit: Programming Teaching! !

The above is the detailed content of Chat MySQL memory management, memory allocator, and operating systems. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:CSDN. If there is any infringement, please contact admin@php.cn delete
图文详解mysql架构原理图文详解mysql架构原理May 17, 2022 pm 05:54 PM

本篇文章给大家带来了关于mysql的相关知识,其中主要介绍了关于架构原理的相关内容,MySQL Server架构自顶向下大致可以分网络连接层、服务层、存储引擎层和系统文件层,下面一起来看一下,希望对大家有帮助。

mysql怎么替换换行符mysql怎么替换换行符Apr 18, 2022 pm 03:14 PM

在mysql中,可以利用char()和REPLACE()函数来替换换行符;REPLACE()函数可以用新字符串替换列中的换行符,而换行符可使用“char(13)”来表示,语法为“replace(字段名,char(13),'新字符串') ”。

mysql的msi与zip版本有什么区别mysql的msi与zip版本有什么区别May 16, 2022 pm 04:33 PM

mysql的msi与zip版本的区别:1、zip包含的安装程序是一种主动安装,而msi包含的是被installer所用的安装文件以提交请求的方式安装;2、zip是一种数据压缩和文档存储的文件格式,msi是微软格式的安装包。

mysql怎么去掉第一个字符mysql怎么去掉第一个字符May 19, 2022 am 10:21 AM

方法:1、利用right函数,语法为“update 表名 set 指定字段 = right(指定字段, length(指定字段)-1)...”;2、利用substring函数,语法为“select substring(指定字段,2)..”。

mysql怎么将varchar转换为int类型mysql怎么将varchar转换为int类型May 12, 2022 pm 04:51 PM

转换方法:1、利用cast函数,语法“select * from 表名 order by cast(字段名 as SIGNED)”;2、利用“select * from 表名 order by CONVERT(字段名,SIGNED)”语句。

MySQL复制技术之异步复制和半同步复制MySQL复制技术之异步复制和半同步复制Apr 25, 2022 pm 07:21 PM

本篇文章给大家带来了关于mysql的相关知识,其中主要介绍了关于MySQL复制技术的相关问题,包括了异步复制、半同步复制等等内容,下面一起来看一下,希望对大家有帮助。

带你把MySQL索引吃透了带你把MySQL索引吃透了Apr 22, 2022 am 11:48 AM

本篇文章给大家带来了关于mysql的相关知识,其中主要介绍了mysql高级篇的一些问题,包括了索引是什么、索引底层实现等等问题,下面一起来看一下,希望对大家有帮助。

mysql怎么判断是否是数字类型mysql怎么判断是否是数字类型May 16, 2022 am 10:09 AM

在mysql中,可以利用REGEXP运算符判断数据是否是数字类型,语法为“String REGEXP '[^0-9.]'”;该运算符是正则表达式的缩写,若数据字符中含有数字时,返回的结果是true,反之返回的结果是false。

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
3 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

SublimeText3 Linux new version

SublimeText3 Linux new version

SublimeText3 Linux latest version

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

WebStorm Mac version

WebStorm Mac version

Useful JavaScript development tools

SublimeText3 English version

SublimeText3 English version

Recommended: Win version, supports code prompts!