MySQL OOM Series 1 Linux Memory Allocation_MySQL

RDS (NetEase Cloud Relational Database Service) has been online for some time, and products have been continuously moved into RDS. During the online operation and maintenance process, we also encountered some things that we had not considered, or had not fully considered. I can share it with you when I have time later.

What I want to mention today is a 4G RDS instance online. An OOM (out of memory) problem occurred, and the MySQL process was directly killed. When explaining this problem, we first need to start with the Linux system memory allocation strategy.
Generally when writing C language programs, we are accustomed to using malloc to dynamically apply for memory space (Java is responsible for memory management by the JVM). The malloc function will apply to the operating system for a continuous memory unit, and then return the starting address of this space. If the malloc function returns null, it means that the system has no memory space to allocate. This is our general thinking, and of course this is indeed correct in some operating systems (Solaris).
But Linux is not like this. Linux's memory allocation adopts a more aggressive allocation strategy. It assumes that the application will not use the memory space immediately after applying for it, so a certain amount of overbooking is allowed when the application really needs it. When using it, the operating system may have become capable of meeting the needs of this application by reclaiming the memory space of other applications. Simply put, it allows the application to apply for more space than the actual allocable space (including physical memory and Swap). more memory, this feature is called OverCommit.
This feature is also configurable in the Linux operating system. You can adjust the OverCommit policy by setting /proc/sys/overcommit_memory to different values.
Overcommit_memory can take 3 values:
0: Default value. The Linux kernel uses some heuristic algorithms to determine whether to overbook and the size of the overbook. Generally, slight overbooking is allowed, some requests that are obviously impossible to provide are rejected, and some rule restrictions are imposed, such as overcommit by different users. The sizes are also different.
1: Allowed, overbooking without restrictions. Of course, this is not infinite, and is also limited by the addressing space. The maximum possible size for 32-bit systems is only 4G, and the maximum for 64-bit systems is about 16T.
2: Forbidden, overbooking is prohibited. The memory that the system can allocate will not exceed swap + actual physical memory * overcommit_ratio. This value can be set through /proc/sys/vm/overcommit_ratio, and the default is 50%.

In order to verify the memory allocation of Linux, we use a small program to test it:

#include <stdio.h>
#include <stdlib.h>
#define MEGABYTE 1024*1024
int main(int argc, char *argv[])
{
    void *myblock = NULL;
    int count = 0;

    while (1)
    {
        myblock = (void *) malloc(MEGABYTE);
        if (!myblock) break;
        printf("Currently allocating %d MB\n", ++count);
    }
    
    exit(0);
}

#include <stdio.h>
#include <stdlib.h>

#define MEGABYTE 1024*1024

int main(int argc, char *argv[])
{
    void *myblock = NULL;
    int count = 0;

    while(1)
    {
        myblock = (void *) malloc(MEGABYTE);
        if (!myblock) break;
        memset(myblock,1, MEGABYTE);
        printf("Currently allocating %d MB\n",++count);
    }
    exit(0);
    
}

After the former applied for memory space through malloc(), it did not use it immediately, while the latter, on the contrary, immediately filled it with 1 after each application. Let's take a look at the results of running the two programs.

This is the result of running on a virtual machine with 1G RAM and 400M Swap. The former applies for far more space than the actual memory, while the latter does not exceed the actual available memory space. This verifies the Linux memory allocation strategy described earlier.
This is a systematic optimization in itself, which is understandable. But we know that any "overbooking" is based on the assumption that there will not be a large number of programs using resources at the same time, which is obviously risky. Therefore, Linux uses an OOM Killer (Out Of Memory killer) mechanism to selectively kill some processes to release some memory before the system's available memory (including Swap) is about to be used up. In the next chapter, we will focus on the mechanism of Linux OOM Killer.