linux - 服务器半夜磁盘使用率超标如何分析？

Question

服务器版本：CentOS Linux release 7.1.1503 (Core)
最近老是半夜收到阿里云的短信提示服务器的磁盘使用率超标了。
使用sar -A看到有异常情况如下。但是具体为啥会超标该怎么继续分析？

12:00:01 AM  pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff
12:10:01 AM      0.09      3.85    220.27      0.00     79.41      0.00      0.00      0.00      0.00
12:20:01 AM      0.01      1.91    206.10      0.00     75.79      0.00      0.00      0.00      0.00
12:30:01 AM      0.01      2.03    206.22      0.00     75.87      0.00      0.00      0.00      0.00
12:40:01 AM      0.00      3.80    210.53      0.00     77.26      0.00      0.00      0.00      0.00
12:50:01 AM      0.00      2.09    206.14      0.00     75.57      0.00      0.00      0.00      0.00
01:00:01 AM      0.00      3.18    206.27      0.00     75.88      0.00      0.00      0.00      0.00
01:10:01 AM      0.00      2.12    220.06      0.00     79.43      0.00      0.00      0.00      0.00
01:20:01 AM      0.00      1.97    206.23      0.00     75.79      0.00      0.00      0.00      0.00
01:30:01 AM      0.01      1.89    206.15      0.00     75.65      0.00      0.00      0.00      0.00
01:40:01 AM      0.00      3.05    220.17      0.00     79.93      0.00      0.00      0.00      0.00
01:50:01 AM      0.00      2.55    206.20      0.00     75.55      0.00      0.00      0.00      0.00
02:00:01 AM      0.00      2.05    206.18      0.00     75.47      0.00      0.00      0.00      0.00
02:10:01 AM      0.00      2.11    220.02      0.00     79.37      0.00      0.00      0.00      0.00
02:20:01 AM      0.00      1.85    206.08      0.00     75.73      0.00      0.00      0.00      0.00
02:30:01 AM      0.00      2.07    206.16      0.00     75.84      0.00      0.00      0.00      0.00
02:40:01 AM      0.00      2.39    210.69      0.00     76.96      0.00      0.00      0.00      0.00
02:50:01 AM      0.00      2.38    206.17      0.00     75.51      0.00      0.00      0.00      0.00
03:00:01 AM      0.00      3.42    206.28      0.00     75.62      0.00      0.00      0.00      0.00
03:10:01 AM      0.00      4.69    219.82      0.00     79.34      0.00      0.00      0.00      0.00
03:20:01 AM      0.00      4.75    217.01      0.00     79.04      0.00      0.00      0.00      0.00
03:30:01 AM     53.62      5.56    207.01      0.00     76.56      0.00      0.00      0.00      0.00
03:40:01 AM    287.65     37.00    245.74      0.15    100.08      0.00      0.00      0.00      0.00
03:50:01 AM    187.61      7.89    208.05      0.00     80.63      0.00      0.00      0.00      0.00
04:00:01 AM     19.75      7.07    217.46      0.00     88.48      0.00      0.00      0.00      0.00
04:10:01 AM    181.71      8.41    212.29      0.00     83.98      0.00      0.00      0.00      0.00
04:20:01 AM    385.70      9.50    222.33      0.03    674.64     34.72      0.00     33.86     97.53
04:30:01 AM    385.34     10.32    209.89      0.00     85.06      0.00      0.00      0.00      0.00
04:40:01 AM    136.85      7.67    211.48      0.00     83.73      0.00      0.00      0.00      0.00
04:50:01 AM    264.71      7.59    207.58      0.00     83.15      0.00      0.00      0.00      0.00
05:00:01 AM    393.60      7.53    210.64      0.00     85.45      0.00      0.00      0.00      0.00
05:10:01 AM    262.00      6.47    222.05      0.00     87.20      0.00      0.00      0.00      0.00
05:20:01 AM      1.09      3.71    206.89      0.00     76.29      0.00      0.00      0.00      0.00
05:30:01 AM      0.31      3.27    206.40      0.00     75.83      0.00      0.00      0.00      0.00
05:40:01 AM      0.00      2.50    210.33      0.00     76.70      0.00      0.00      0.00      0.00
05:50:01 AM      0.00      3.83    206.34      0.00     75.67      0.00      0.00      0.00      0.00
06:00:01 AM      0.00      2.37    206.22      0.00     75.64      0.00      0.00      0.00      0.00
06:10:01 AM      0.00      2.13    218.19      0.00     78.98      0.00      0.00      0.00      0.00
06:20:01 AM      0.00      1.91    206.10      0.00     75.84      0.00      0.00      0.00      0.00
06:30:01 AM      0.00      1.97    206.19      0.00     75.81      0.00      0.00      0.00      0.00
06:40:01 AM      0.01      2.25    210.30      0.00     77.00      0.00      0.00      0.00      0.00
06:50:01 AM      0.00      1.85    206.11      0.00     76.18      0.00      0.00      0.00      0.00
07:00:01 AM      0.00      2.11    206.10      0.00     75.74      0.00      0.00      0.00      0.00
07:10:01 AM      0.00      1.94    218.21      0.00     78.92      0.00      0.00      0.00      0.00
07:20:01 AM      0.00      1.90    206.10      0.00     75.53      0.00      0.00      0.00      0.00
07:30:01 AM      0.07      2.03    209.91      0.00     78.93      0.00      0.00      0.00      0.00
07:40:01 AM      0.10      2.25    218.15      0.00     77.00      0.00      0.00      0.00      0.00
07:50:01 AM      0.00      3.33    196.47      0.00     72.36      0.00      0.00      0.00      0.00
08:00:01 AM      1.12      3.37    209.59      0.01     76.83      0.00      0.00      0.00      0.00
08:10:01 AM      0.00      1.97    221.85      0.00     79.84      0.00      0.00      0.00      0.00
08:20:01 AM      0.00      3.62    207.75      0.00     75.96      0.00      0.00      0.00      0.00
08:30:01 AM      0.00      2.35    211.21      0.00     78.11      0.00      0.00      0.00      0.00
08:40:01 AM      0.00      2.25    200.32      0.00     86.28      0.00      0.00      0.00      0.00
08:50:01 AM     21.21      4.25    271.61      0.09     99.59      0.00      0.00      0.00      0.00
09:00:01 AM      9.65      3.29    414.17      0.05    167.70      0.00      0.00      0.00      0.00
09:10:01 AM      0.83      2.37    261.60      0.00    107.26      0.00      0.00      0.00      0.00

12:00:01 AM       tps      rtps      wtps   bread/s   bwrtn/s
12:10:01 AM      0.72      0.01      0.71      0.17      7.71
12:20:01 AM      0.32      0.00      0.32      0.01      3.83
12:30:01 AM      0.32      0.00      0.32      0.01      4.07
12:40:01 AM      0.65      0.00      0.65      0.00      7.60
12:50:01 AM      0.33      0.00      0.33      0.00      4.17
01:00:01 AM      0.57      0.00      0.57      0.00      6.36
01:10:01 AM      0.35      0.00      0.35      0.00      4.24
01:20:01 AM      0.32      0.00      0.32      0.00      3.93
01:30:01 AM      0.31      0.00      0.31      0.01      3.79
01:40:01 AM      0.51      0.00      0.51      0.00      6.09
01:50:01 AM      0.43      0.00      0.43      0.00      5.09
02:00:01 AM      0.35      0.00      0.35      0.00      4.09
02:10:01 AM      0.35      0.00      0.35      0.00      4.21
02:20:01 AM      0.31      0.00      0.31      0.00      3.69
02:30:01 AM      0.31      0.00      0.31      0.00      4.13
02:40:01 AM      0.36      0.00      0.36      0.00      4.77
02:50:01 AM      0.40      0.00      0.40      0.00      4.76
03:00:01 AM      0.59      0.00      0.59      0.00      6.84
03:10:01 AM      0.89      0.00      0.89      0.00      9.37
03:20:01 AM      0.89      0.00      0.89      0.00      9.51
03:30:01 AM      2.23      1.64      0.59    107.24     11.12
03:40:01 AM     11.69      9.81      1.88    574.98     74.00
03:50:01 AM      7.83      7.13      0.70    375.55     15.77
04:00:01 AM      2.13      1.49      0.64     39.51     14.15
04:10:01 AM      8.36      7.63      0.74    363.42     16.81
04:20:01 AM     14.69     13.78      0.91    771.41     19.00
04:30:01 AM     14.70     13.66      1.04    770.68     20.64
04:40:01 AM      8.09      7.45      0.64    273.70     15.33
04:50:01 AM     11.26     10.64      0.62    529.42     15.19
05:00:01 AM     13.89     13.27      0.62    787.20     15.05
05:10:01 AM      9.42      8.83      0.59    523.99     12.93
05:20:01 AM      0.74      0.08      0.66      2.17      7.43
05:30:01 AM      0.60      0.02      0.59      0.63      6.55
05:40:01 AM      0.44      0.00      0.44      0.00      5.00
05:50:01 AM      0.71      0.00      0.71      0.00      7.65
06:00:01 AM      0.41      0.00      0.41      0.00      4.73
06:10:01 AM      0.36      0.00      0.36      0.00      4.25
06:20:01 AM      0.33      0.00      0.33      0.00      3.83
06:30:01 AM      0.34      0.00      0.34      0.00      3.95
06:40:01 AM      0.38      0.00      0.37      0.03      4.51
06:50:01 AM      0.32      0.00      0.32      0.00      3.71
07:00:01 AM      0.36      0.00      0.36      0.00      4.21
07:10:01 AM      0.33      0.00      0.33      0.00      3.88
07:20:01 AM      0.33      0.00      0.33      0.00      3.80
07:30:01 AM      0.36      0.01      0.34      0.15      4.07
07:40:01 AM      0.36      0.00      0.36      0.20      4.49
07:50:01 AM      0.58      0.00      0.58      0.00      6.65
08:00:01 AM      0.65      0.05      0.60      2.24      6.75
08:10:01 AM      0.34      0.00      0.34      0.00      3.95
08:20:01 AM      0.67      0.00      0.67      0.00      7.24
08:30:01 AM      0.40      0.00      0.40      0.00      4.69
08:40:01 AM      0.38      0.00      0.38      0.00      4.51
08:50:01 AM      1.44      0.87      0.57     42.43      8.51
09:00:01 AM      0.93      0.42      0.50     19.31      6.57

Answer 1

You can use a tool called atop, which can view and record the process list of disk I/O usage from high to low at a certain moment.
Service /etc/init.d/atop will run /etc/atop/atop.daily
The log collection interval (INTERVAL) is defined in atop.daily as 600 seconds.
Log storage location /var/log/atop/atop_YYYYMMDD is binary content. You can view it like this:
atop -r /var/log/atop/atop_YYYYMMDD Press the t key to switch to the next collection point. , press d to check the disk I/O status.

Compile and install the latest version of atop:

http://www.atoptool.nl/downloadatop.php
wget http://www.atoptool.nl/download/atop-2.1-1.tar.gz
tar xzf atop-2.1-1.tar.gz && cd atop-2.1
make && sudo make install
atop

Use atop (written in C) or iotop (written in Python) to check the I/O status of a specific process, and use iostat (sysstat) to check the global I/O load of the system.
http://www.atoptool. nl/screen...
atop interactive command:
t (manual refresh, the default refresh interval is 10 seconds, i can modify the time interval) g (default output) d (disk) m (memory)
s (scheduling) v (variable) c (command line) p (process statistics) u (user statistics) n (network, requires kernel module netatop support)
C (sort by CPU) M (sort by memory MEM) D(Sort by disk DSK) N(Sort by network NET)

atop -d View system disk overview:
The data above is the system overview: CPU and its core/MEM (memory)/SWP (swap space)/PAG (page swap)/LVM (logical volume)/DSK ( Disk)/NET (Network)
The data below is the specific process: PID RDDSK (read) WRDSK (write) WCANCL DSK (disk usage of the process) CMD

atop can also display the disk I/O and network I/O status of the system and process, which is better than top. It even developed a kernel module netatop in order to monitor the network I/O of specific processes.