Parsing the CentOS service program performance evaluation document

1 Overview

1.1 Factors affecting the performance of Linux service programs

CPU, memory, disk I/O bandwidth, network I/O bandwidth

1.2 Performance Evaluation

CPU: user% sys%<70%; The percentage of execution time of the program in user mode and kernel mode.

Memory: Swap In (si) = 0; Swap Out (so) = 0; Subject to not using the swap partition. If the swap partition is frequently used, the memory may not be enough.

Hard disk: iowait % < 20%;

Network: As long as you have enough bandwidth, use it to your heart's content. If the network card bandwidth is reached, the Linux system will have no pressure.

Among them: %user: Indicates the percentage of time the CPU is in user mode.

%sys: Indicates the percentage of time the CPU is in kernel mode.

%iowait: Indicates the percentage of time the CPU waits for input and output to be completed.

Swap in: si, which means page import of virtual memory, that is, swapping from SWAP DISK to RAM

Swap out: so, which means page export of virtual memory, that is, swapping from RAM to SWAP DISK.

1.3 Performance Analysis Tool

Commonly used system commands: top, free, ps, uptime, iotop, vmstat, iostat, dstat, sar.
Usage: The top command grasps the overall situation, and uses specific commands for in-depth analysis

Common combination methods:

(1) CPU bottleneck: top, vmstat, iostat, sar –u, sar -q

(2) Memory bottleneck: free, vmstat, sar -B, sar -r, sar -W

(3) Disk I/O bottleneck: iotop, iostat, sar -b, sar –u, sar -d

(4) Network bottleneck: dstat

2 top

2.1 Function

Provides real-time status monitoring of system processors, memory, tasks, etc.; this command can sort tasks according to CPU usage and memory usage; TOP is a dynamic display process that can continuously refresh the current status through user keystrokes. The refresh interval can also be specified at startup.

2.2 Command output diagram

top - 10:16:29 up 38 days, 15:48, 5 users, load average: 0.04, 0.10, 0.05

Tasks: 569 total, 2 running, 562 sleeping, 0 stopped, 5 zombie

Cpu(s): 2.6%us, 1.3%sy, 0.4%ni, 95.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 3839112k total, 3151560k used, 687552k free, 302944k buffers

Swap: 6078456k total, 631852k used, 5446604k free, 348548k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME COMMAND                                                              

10603 zhixiang 20 0 1225m 284m 30m S 4.6 7.6 271:48.24 vs_exe                                                        

2473 xulun 20 0 118m 43m 6932 S 3.0 1.2 964:14.99​

1412 xulun 30 10 232m 12m 5216 S 2.3 0.3 0:04.59 floaters                                                                                                            

#14823 chujie 20 0 1112m 247m 17m S 2.0 6.6 363:51.38 vs_exe                                                      ​

17772 sihao 20 0 1101m 202m 10m S 1.7 5.4 631:21.17 vs_exe                                                                  

11054 sihao 20 0 906m 65m 9412 S 1.0 1.7 410:06.55 vs_exe                                                          

20782 yongtao 20 0 1049m 173m 9996 S 1.0 4.6 196:05.10 vs_exe                                                

14637 chujie 20 0 1274m 132m 2388 S 0.7 3.5 63:20.18 KugooPopMsgServ                                                                                                        

#1439 yanyun 20 0 15428 1600 940 R 0.3 0.0 0:00.01 top                                                                        

3491 zhixiang 20 0 129m 55m 17m S 0.3 1.5 10:57.25

1 root 20 0 19344 1200 984 S 0.0 0.0 0:00.89 init

2.3 Command output analysis

Top outputs a lot of information and basically covers all performance indicators. The first five lines are the statistical information area, which represents the overall statistical information of the system, including: system load, tasks, CPU, memory, etc.; the following are each process Related Information.

The first line is the task queue information, which is the same as the execution result of the uptime command

[yanyun@~/test]$ uptime

10:20:17 up 38 days, 15:52, 5 users, load average: 0.00, 0.04, 0.03

The content is as follows: 10:20:17: Current time

38 days, 15:52：System running time

5 users: Number of currently logged in users

load average: 0.00, 0.04, 0.03: System load, that is, the average length of the task queue. The three values ​​are the average values ​​from 1 minute, 5 minutes, and 15 minutes ago to now.

The second line is process information

The content is as follows: 569 total: total number of processes

2 running,: Number of running processes

562 sleeping: Number of sleeping processes

0 stopped: Number of stopped processes

5 zombie: Number of zombie processes

The third line is CPU information

The content is as follows: 2.6%us: The percentage of CPU occupied by user space

1.3%sy: Percentage of CPU occupied by kernel space

0.4%ni: The percentage of CPU occupied by processes that have changed priorities in the user process space

95.7%id: Idle CPU percentage

0.0%wa: Percentage of CPU time waiting for input and output

The fourth and fifth lines are memory information; the command output is the same as free

[yanyun@~]$ free

total used used free shared buffers cached

Mem: 3839112 3256976 582136 0 143664 444992

-/ buffers/cache: 2668320 1170792

Swap: 6078456 574772 5503684

The content is as follows: Mem:3839112k total: total physical memory

3151560k used: Total amount of physical memory used

687552k free：Total amount of free memory

302944k buffers: Amount of memory used for buffering

Swap: 6078456k total：Total amount of swap area

631852k used: Total amount of swap area used

5446604k free: Total amount of free swap area

348548k cached: Total amount of cache.

Note: buffer: can be thought of as a buffer written to disk;

Cache: Read the disk cache.

The principle of using memory in the Linux system is: if you don’t use it, it will be used in vain; cache things as much as possible, so the free memory is often very small, but the cache is very large; the Linux system will regularly start the kernel thread kswapd for caching Recycle.

The following is to display information related to each process

%CPU: The percentage of CPU time occupied since the last update

TIME: Total CPU time used by the process

%MEM: Percentage of physical memory used by the process

VIRT: The total amount of virtual memory used by the process, unit kb

RES: The size of the physical memory used by the process and not swapped out, in kb. RES=CODE DATA

SHR: Shared memory size, unit kb

S: Process status. (D=uninterruptible sleep state R=run S=sleep T=track/stop Z=zombie process)

2.4 Common options

top [-] [d][p][M][P]

Parameter Description:

d: Specify the time interval between every two screen information refreshes. (top –d 1: refresh once per second)

p: Specify the process ID to monitor only a certain process. (top –d 1234: Only view process information with pid 1234)

k: Terminate a process. Top runtime parameters, the system will prompt the user to enter the PID of the process that needs to be terminated, and what kind of signal needs to be sent to the process. Use signal 9 to force the process to end.

M: Sort based on resident memory size.

P: Sort according to CPU usage percentage.

Note: Press the numeric key ‘1’ during command line execution to view relevant information about each core of the CPU.

Tasks: 564 total, 3 running, 556 sleeping, 0 stopped, 5 zombie

Cpu0 : 2.9%us, 2.9%sy, 0.0%ni, 94.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu1 : 2.0%us, 2.0%sy, 0.0%ni, 96.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 3839112k total, 3445296k used, 393816k free, 48180k buffers

Swap: 6078456k total, 553876k used, 5524580k free, 976128k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME COMMAND                                                            

14823 chujie 20 0 1112m 245m 16m R 3.0 6.5 373:07.82 vs_exe                                                        

5589 sihao 20 0 1019m 267m 29m R 2.0 7.1 2:24.80 vs_exe                                                      

5674 zhixiang 20 0 1103m 253m 37m S 2.0 6.8 4:17.89 vs_exe

3 iotop：

3.1 Function

is a top-like tool used to monitor the disk I/O usage of each thread,

Note: This command needs to be installed by yourself (yum install iotop)

3.2 Command output diagram

Total DISK READ: 50.23 M/s | Total DISK WRITE: 34.25 K/s

TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND                                                          

61524 be/4 root 47.65 M/s 0.00 B/s 0.00 % 37.83 % ./relay_server

61539 be/4 root 121.77 K/s 0.00 B/s 0.00 % 26.73 % ./relay_server

61544 be/4 root 700.15 K/s 0.00 B/s 0.00 % 24.89 % ./relay_server

61543 be/4 root 528.92 K/s 0.00 B/s 0.00 % 21.29 % ./relay_server

61541 be/4 root 494.67 K/s 0.00 B/s 0.00 % 21.22 % ./relay_server

61540 be/4 root 323.44 K/s 0.00 B/s 0.00 % 8.62 % ./relay_server

61542 be/4 root 468.04 K/s 0.00 B/s 0.00 % 8.13 % ./relay_server

480 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.02 % [jbd2/sda2-8]

1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init

2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]

3 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0]

3.3 Command output analysis

first row:

Total DISK READ: 50.23 M/s: The amount of data read from the disk per second

Total DISK WRITE: 34.25 K/s: The amount of data written to the disk per second

The following is the IO situation of each thread:

is relatively simple; just explain IO: it has the same meaning as top's wa, but it is the wa of a thread represented here.

3.4 Common options

iotop [-] [d] [p]

Parameter Description:

d: Specify the time interval between every two screen information refreshes. (iotop –d 1: refresh once per second)

p: Specify the thread ID to monitor only a certain thread. (iotop –d 1234: Only view process information with pid 1234)

Note: Press the letter ‘o’ during command line execution to view only threads with IO.

pstree -p: View the process tree and output the relationship between processes.

ps –eLf: View threads. Advanced version of ps –ef

4 vmstat:

4.1 Function

Tool to check virtual memory usage

4.2 Command output diagram

[yanyun@~/test]$ vmstat

procs -----------memory---------- ---swap-- -----io---- --system-- --- --cpu-----

r b swpd free buff cache si so bi bo in cs us sy id wa st

0 0 631376 834896 15108 452024 0 0 4 4 1 2 2 1 97 0 0

4.3 Command output analysis

Memory: The fourth and fifth lines of information are the same as top.

swap: Memory to swap partition swap-in and swap-out rate.

io: read and write disk rate

system in: Number of interrupts per second, including clock interrupts

system cs: The number of environment (context) switches per second; frequent switching has an impact on system performance. Linux provides thread affinity for the CPU, which can bind a thread to a core for running.

cpu: Same as the third line of top output

Note: The swap item is not zero, and the swap partition is used. The memory may have been used up and has become a system bottleneck.

4.4 Common options

vmstat 1 10: Refresh once per second, refresh and exit ten times in total.

Note: The following options are all for professional viewing of information on a specific item. Some output a lot of information. Google it when you need it!

-a: Display active and inactive memory

-f: Display the number of forks since system startup

-m: display slabinfo

-s: Display memory-related statistical information and the number of various system activities.

-d: Display disk related statistics.

-p: Display the statistics of the specified disk partition

5 iostat

5.1 Function

Used to output statistics related to CPU and disk I/O

5.2 Output result diagram

[root@ShanWei_119_134_255_208 ~]# iostat -x

Linux 2.6.32-279.el6.x86_64 (ShanWei_119_134_255_208) 07/30/2013 _x86_64_ (16 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle

2.63 0.00 2.25 5.98 0.00 89.14

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util

sda 195.42 163.72 194.33 126.28 28464.35 2319.94 96.02 1.16 3.63 1.41 45.30

sdb 0.33 48.97 122.71 2.70 29603.37 413.38 239.34 0.27 2.15 1.00 12.52

5.3 Command output explanation

avg-cpu segment:

%user: Percentage of CPU used when running at user level.

%nice: The percentage of CPU used by nice operations.

%sys: The percentage of CPU used when running at the system level (kernel).

%iowait: The percentage of CPU occupied when the CPU is waiting for hardware I/O.

%idle: The percentage of CPU idle time.

Device segment:

tps: Number of I/O requests sent per second

Blk_read /s: Number of blocks read per second

Blk_wrtn/s: Number of blocks written per second

Blk_read: Total number of blocks read

Blk_wrtn: Total number of blocks written

-x option can view the following more information

rrqm/s: How many read requests related to this device were merged per second

wrqm/s: How many write requests related to this device were merged per second

rsec/s: Number of sectors read per second

wsec/: Number of sectors written per second

avgrq-sz: Average data size (sectors) per device I/O operation

avgqu-sz: Average I/O queue length

await: average waiting time for each device I/O operation (unit is milliseconds)

svctm: Average service time of each device I/O operation (unit is milliseconds)

%util: All processing IO time within the statistical time, so this parameter implies the busyness of the device

5.4 Common options

iostat -d 2 6: Display device statistics every 2 seconds. A total of 6 output times.

-x: View device usage and response time

6 dstat

6.1 Function

is an all-round system information statistics tool that can only monitor the entire system but cannot conduct in-depth analysis of a certain process or program; monitoring items include: cpu, disk, memory, network card, process, system (color interface It’s rare under Linux!)

Note: This command needs to be installed by yourself (yum install dstat)

6.2 Command output diagram

[root@ShanWei_119_134_255_208 ~]# dstat -cdlmnpy

----total-cpu-usage---- -dsk/total- ---load-avg--- ------memory-usage----- -net/total- - --procs--- ---system--

usr sys idl wai hiq siq| read writ| 1m 5m 15m | used buff cache free| recv send|run blk new| int csw

3 2 89 6 0 0| 28M 1367k|5.41 5.07 4.87|4019M 172M 58.5G 267M| 0 0 | 0 0.0 1.3|8608 35k

2 1 85 12 0 0| 21M 828k|5.41 5.07 4.87|4020M 172M 58.6G 243M| 191k 15M| 0 6.0 0|5711 18k

1 1 80 18 0 0|5828k 4804k|5.41 5.07 4.87|4018M 172M 58.6G 260M| 190k 17M|2.0 3.0 0|5802 12k

6.3 Command output explanation

Basically explained before...

6.4 Common options

dstat –cdlmnpsy: cdlmnpsy these options basically cover the commonly used ones, of course there are many dstat – help.

7 sar

7.1 Function

System Activity Reporter system activity report is one of the most comprehensive system performance analysis tools currently on Linux. It can report system activities from many aspects, including: file reading and writing, system call usage, Disk I/O, CPU efficiency, memory usage, process activity and IPC-related activities, etc.

7.2 Command output diagram

[root@localhost ~]# sar -r 1 100

Linux 2.6.32-220.el6.x86_64 (localhost.localdomain) 07/30/2013 _x86_64_ (8 CPU)

03:17:59 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit

03:18:00 PM 4988488 3062224 38.04 639136 2115404 72404 0.44

03:18:01 PM 4984464 3066248 38.09 639136 2115404 98060 0.60

03:18:02 PM 4985152 3065560 38.08 639136 2115420 97972 0.60

03:18:03 PM 4985400 3065312 38.08 639136 2115420 97972 0.60

[root@localhost ~]# sar -B 1 100

Linux 2.6.32-220.el6.x86_64 (localhost.localdomain) 07/30/2013 _x86_64_ (8 CPU)

03:19:09 PM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff

03:19:10 PM 0.00 157.14 45415.48 0.00 15541.67 0.00 0.00 0.00 0.00

03:19:11 PM 0.00 0.00 40.40 0.00 75.76 0.00 0.00 0.00 0.00

03:19:12 PM 0.00 0.00 5542.34 0.00 1584.68 0.00 0.00 0.00 0.00

[root@localhost ~]# sar -b 1 100

Linux 2.6.32-220.el6.x86_64 (localhost.localdomain) 07/30/2013 _x86_64_ (8 CPU)

03:19:38 PM tps rtps wtps bread/s bwrtn/s

03:19:39 PM 12.50 0.00 12.50 0.00 181.82

03:19:40 PM 0.00 0.00 0.00 0.00 0.00

03:19:41 PM 7.14 0.00 7.14 0.00 128.57

[root@localhost ~]# sar -W 1 100

Linux 2.6.32-220.el6.x86_64 (localhost.localdomain) 07/30/2013 _x86_64_ (8 CPU)

03:20:10 PM pswpin/s pswpout/s

03:20:11 PM 0.00 0.00

03:20:12 PM 0.00 0.00

03:20:13 PM 0.00 0.00

[root@localhost ~]# sar -d 1 100

Linux 2.6.32-220.el6.x86_64 (localhost.localdomain) 07/30/2013 _x86_64_ (8 CPU)

03:20:48 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util

03:20:49 PM dev8-0 7.69 0.00 79.12 10.29 0.00 0.14 0.14 0.11

03:20:49 PM dev8-16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

03:20:49 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util

03:20:50 PM dev8-0 10.47 0.00 148.84 14.22 0.08 7.22 7.11 7.44

03:20:50 PM dev8-16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

7.3 Command output explanation

kbmemfree: This value is basically the same as the free value in the free command, so it does not include buffer and cache space.

kbmemused: This value is basically the same as the used value in the free command, so it includes buffer and cache space.

%memused: This value is a percentage of kbmemused and total memory (excluding swap).

kbbuffers and kbcached: These two values ​​​​are the buffer and cache in the free command.

kbcommit: Ensure the memory required by the current system, that is, the memory required to ensure no overflow (RAM swap).

%commit: This value is a percentage of kbcommit and total memory (including swap).

pgpgin/s: Indicates the number of bytes (KB) replaced from disk or SWAP to memory per second

pgpgout/s: Indicates the number of bytes (KB) replaced from memory to disk or SWAP per second

fault/s: The number of page faults generated by the system per second, that is, the sum of major page faults and minor page faults (major minor)

majflt/s: The number of main missing pages generated per second.

pgfree/s: The number of pages placed in the free queue per second

pgscank/s: The number of pages scanned by kswapd per second

pgscand/s: Number of pages directly scanned per second

pgsteal/s: The number of pages that are cleared from the cache to meet memory requirements per second

%vmeff: The percentage of pages cleared per second (pgsteal) to the total scanned pages (pgscank pgscand)

tps: Total I/O transfer volume of physical devices per second

rtps: The total amount of data read from the physical device per second

wtps: The total amount of data written to the physical device per second

bread/s: The amount of data read from the physical device per second, in blocks/s

bwrtn/s: The amount of data written to the physical device per second, in blocks/s

pswpin/s: The number of swap pages swapped in by the system per second

pswpout/s: The number of swap pages swapped out by the system per second

7.4 Common options

sar –[u][r][B][b][q][W][d]

Parameter Description:

-u: CPU resource monitoring

-r: Memory and swap space monitoring

-B: Memory paging monitoring

-b: IO and transfer rate monitoring

-q: Process queue system load monitoring

-W: System swap partition activity monitoring

-d: Device usage monitoring

8 tmpfs

8.1 Definition

The tmpfs file system is a memory-based file system mounted under /dev/shm.

8.2 Features

Change the size dynamically; access quickly and completely reside in RAM; of course, it will disappear after the system restarts.

8.3 Usage

1. The POSIX standard shared memory under Linux is implemented based on this file system (there is also a set of System V standard shared memory implementation methods);

2. Files created under /dev/shm will use tmpfs directly; so if you do not need to consider the loss of data after the machine restarts, put the data here to release your disk pressure!

3. The default size of tmpfs is half of the physical memory. To adjust the maximum capacity to 40G and the number of file nodes to 1000000, you can use the following command:

mount -o size=40G -o nr_inodes=1000000 -o noatime,nodiratime -o remount /dev/shm

4. If you need to permanently modify the size of the tmpfs file system, you need to modify /etc/fstab

9 crontab：

9.1 Function

Schedule the execution of some commands regularly, which is equivalent to a system-level timer. There are some files named after user names in /var/spool/cron/, which represent the scheduled execution tasks of this user.

9.2 Configuration method

Format description:

* * * * * /command path

The first five fields can take integer values, specifying when to start work (minutes, hours, dates, months, weeks), the sixth field is a string, scripts, programs, etc. that are due to be executed

43 21 * * * Executed at 21:43 every day

0 17 * * 1 Executed every Monday at 17:00

0,10 17 * * 0,2,3 Executed every Sunday, Tuesday and Wednesday at 17:00 and 17:10

0-10 17 1 * * Execute every 1 minute from 17:00 to 7:10 on the 1st of every month

0 0 1,15 * 1 Executed at 0:00 on the 1st, 15th and 1st of every month

42 4 1 * * Executed at 4:42 on the 1st of every month

0 21 * * 1-6　 Execution at 21:00 from Monday to Saturday

0,10,20,30,40,50 * * * * Execute every 10 minutes

*/10 * * * * Execute every 10 minutes

* 1 * * * Execute every 1 minute from 1:0 to 1:59

0 1 * * *　　　　 1:00 Execute

0 */1 * * * Execute every 0 minutes and every hour

0 * * * * Execute every 0 minutes and every hour

2 8-20/3 * * * 8:02,11:02,14:02,17:02,20:02 Execute

30 5 1,15 * * Execution at 5:30 on the 1st and 15th

The following configuration indicates: execute this script/opt/clear_old_file.sh at 3:00 every day;

The function of this script is: delete files in the two directories /data1 /data2 that have not been accessed within five days

0 3 * * * (cd /opt/ && ./clear_old_file.sh > /dev/null &)

[root@ShanWei_119_134_255_208 ~]# cat /opt/clear_old_file.sh

#!/bin/sh

find /data1/* -type f -atime 5 -exec rm {} \;

find /data2/* -type f -atime 5 -exec rm {} \;

exit 0

10 proc: To be continued…

The

/proc file system is unique to GNU/Linux. It is a virtual file system that resides entirely in RAM, so all files in this directory do not consume disk space. Through it, you can easily understand the kernel information, hardware information, etc. in the system; you can also configure the parameters of the system kernel through it. Many commands actually just collect information from files in /proc and organize it into their own format for display; commands like the ones introduced above basically do this.

The above is the detailed content of Parsing the CentOS service program performance evaluation document. For more information, please follow other related articles on the PHP Chinese website!