Five minutes of technical fun | A brief analysis of Linux Cgroup hierarchical rules

Part 01 cgroup overview

##cgroup is Control Groups The abbreviation of , is a mechanism provided by the Linux kernel that can limit, isolate and count physical resources (such as CPU, memory, device IO, etc.) for processes or process groups. The user space management of cgroup is realized through the cgroup file system. Thanks to the virtual file system of Linux, the details of the file system are hidden, and the user realizes the use of this function through the relevant control files.

cgroup was introduced by Google during the 2.6 kernel period. It is the technical basis for resource virtualization in the Linux kernel and is the basis for LXC (Linux Containers) and Docker containers. technology cornerstone. There are the following related concepts in cgroup:

• Task: An alias for the process;
• Control group: According to a certain A set of processes divided by a standard. Resource control in Cgroup is implemented in units of control groups. A process can be added to a control group or migrated from one process group to another. Processes in a process group can use resources allocated by cgroups in control group units and are subject to resource limits set by cgroup in control group units.
• Hierarchy: Control the hierarchical relationship of the group, using the tree Organized in a structural manner, the control group of the child node inherits the resource setting attributes of the parent node.
• Subsystem: A subsystem is a resource control For example, the CPU subsystem can control the CPU usage time allocation of the process, as shown in Figure 1. A subsystem must be attached to a level to function. After a subsystem is attached to a certain level, all control groups on this level are controlled by this subsystem.

##Part 02 cgroup subsystem

## The cgroup subsystem is related to the kernel version. With the iteration of the kernel, There are more and more resources that can be restricted, generally including the following subsystems.

➤

##blkio：Set restrictions on input/output access to block devices, such as physical devices (disk, SSD, USB, etc.). ➤

cpu：Limit the cpu usage of the process, involving cpu scheduling time slice allocation. ➤

##cpuacct：Automatically generate a cpu report used by tasks in cgroup. ➤

##cpuset：Assign independent CPU (multi-core systems) and memory nodes to tasks in cgroup.

➤ devices: Allow or deny task access in the cgroup equipment.

➤ freezer：Suspend or resume tasks in cgroup.

➤ ##memory：Set memory limits used by tasks in a cgroup and automatically generate reports on memory resources used by those tasks.

➤ ##net_cls：Marking network packets with class identifiers allows the Linux wandering control program to identify packets generated from specific cgroups.

➤ ns：namespace subsystem.

Part 03

cgroup hierarchy rules

Combined with the cgroup hierarchy, it can be understood as a tree. Each node of the tree is a process group, and each tree is associated with one or more subsystems. In a tree, all processes in the Linux system will be included, but each process can only belong to one node (process group). There can be many cgroup trees in the system, and each tree is associated with a different subsystem. A process can belong to multiple trees, that is, a process can belong to multiple process groups, but these process groups are associated with different subsystems. Currently, Linux can build up to twelve cgroup trees, and each tree is associated with a subsystem. Of course, you can also build only one tree and then associate this tree with all subsystems. When a cgroup tree is not associated with any subsystem, it means that the tree only groups processes. As for what to do based on the grouping, it will be decided by the application itself. Systemd is such an example.

There are four rules for the composition of the level, which are described as follows:

Rule 1: A single hierarchy can have one or more subsystems. As shown in Figure 1, the /cpu_memory_cg level sets up two subsystems, cpu and memory, for cgroup1 and cgroup2.

##Figure 1 Hierarchy Rule 1

Rule 2: If any subsystem is already attached to one level, they cannot be attached to the structure of another level. As shown in Figure 2, cpu_cg at level A first manages the cpu subsystem, then cpu_mem_cg at level B cannot manage the cpu subsystem.

Figure 2 cgroup hierarchy rule 2

Rule 3: Every time a new hierarchy is created on a system, all tasks on the system are initially members of that hierarchy's default cgroup (called the root cgroup). For any single hierarchy created, each task on the system can be a cgroup member in that hierarchy. A task can be in multiple cgroups, as long as each of those cgroups is in a different subsystem hierarchy. Once a task becomes a member of the second cgroup in the same hierarchy, it will be deleted from the first cgroup in the hierarchy. That is, two unrelated cgroups in the same hierarchy will never have the same task. That is to say, there can only be one way to restrict a certain type of cgroup subsystem for a certain process. When you create the first hierarchy, every task on the system is a member of at least one cgroup (the root cgroup), so when using cgroups, every system task is always in at least one cgroup, as shown in Figure 3.

Figure 3 cgroup hierarchy rule 3

Rule 4: Any process spawned on the system creates a child process (or thread). The child process automatically inherits the cgroup membership of its parent, but can be moved to other cgroups as needed. After the move, the parent and child processes are completely independent, as shown in Figure 4.

Figure 4 cgroup hierarchy rule 4

Part 04 cgroup hierarchical relationship analysis

We start from the perspective of the process and combine the data structure in the source code to analyze the relationship between cgroups related data. First of all, in Linux, the data structure of the management process is task_struct, in which the members related to cgroups are as follows:

The cgroup points to a css_set structure, which stores cgroups information related to the process. cg_list is a process linked list using the same css_set. The css_set structure is as follows:

## structure The element information of the body is explained as follows:

• refcount is the reference count of css_set, which can be shared by multiple processes, as long as the cgroups information of these processes is the same . For example, processes in the same cgroup in all created hierarchies.
• hlist is used to build all css_sets into a hash table, and the kernel can quickly find specific css_sets.
• #tasks links all processes that reference this css_set into a linked list.
• cg_links points to a linked list composed of struct cg_group_link
• subsys is an array of pointers , stores a set of pointers to cgroup_subsys_state. A cgroup_subsys_state is information related to a process and a specific subsystem. Through this pointer, the process can obtain the corresponding cgroups control information.

Next let’s take a look at the cgroup_subsys_state structure:

##The cgroup pointer in the structure points to a cgroup structure, and the process is affected by the subsystem Resource control is actually achieved by adding a specific cgroup subsystem, because the cgroup is on a specific level, and the subsystem is attached to the level.

Let’s take a look at the structure of cgroup,

• sibling, children The three linked lists with parent are responsible for connecting cgroups at the same level into a tree.
• #susys is the subsystem pointer array described previously.
• root points to a cgroupfs_root structure, which is the structure corresponding to the level where the cgroup is located.
• root->top_cgroup points to the root cgroup of the current level, which is the cgroup automatically created at the Fantasy Sword level. You can get the root cgroup of the hierarchy through cgroup->root->top_cgroup.
• css_sets points to a linked list of cg_cgroup_link, which is consistent with cg_links in css_set.

##For the sake of reason To understand the relationship between css_set and cgroup, we also need to analyze the cg_cgroup_link structure of the middle layer. The structure data is as follows:

The data in the structure is described as follows:

cgrp_link_list is linked to cgroup- >The linked list pointed to by css_sets.

cgrp points to the group related to this cg_cgroup_link.

##cg_link_list is linked to the linked list pointed to by css_set->cg_links.

cg points to the css_set related to cg_cgroup_link.

It can be seen that cgroup and css_set are actually a many-to-many relationship, and an intermediate structure needs to be added to combine the two, cgrp and cg in cg_group_link The element is the joint, and the two linked lists cgrp_link_list and cg_link_list are the attached cgroup and css_set entities, which facilitate polling.

It can be seen from the hierarchical rules of cgroup that a group of processes can belong to cgroups that are not at the same level. Combined understanding, a css_set stores a group of Information related to each subsystem of the process root. The subsystems come from different cgroup levels, so the cgroup_subsys_state stored in a css_set can correspond to multiple cgroups. On the other hand, the cgroup level also stores a set of cgroup_subsys_state, which is obtained from the subsystem attached to the level where the cgroup is located. A cgroup can have multiple processes, and the css_set of the process is not necessarily the same, because the process may use multiple levels. Therefore, a cgroup also needs to correspond to multiple css_sets. Figure 5 describes the many-to-many hooking relationship in detail.

Figure 5 Process and cgroup many-to-many relationship diagram

Part 05 Conclusion

This article is based on the concept of cgroup. The many-to-many relationship between it and the process is dismantled, and its specific code implementation is analyzed from the hooking of variables in the relevant structure, hoping to help readers have a better understanding of the cgroup hierarchical relationship and usage.

The above is the detailed content of Five minutes of technical fun | A brief analysis of Linux Cgroup hierarchical rules. For more information, please follow other related articles on the PHP Chinese website!