事情经过
昨天早上,打开电脑发现自己的博客网站打开不了,准备远程登录服务器查看问题,发现服务器远程不上。没办法,登录阿里云后台,重启服务器。重启完成后,网站能正常打开,所以当时就不以为然,以为阿里云那边是不是出了什么毛病。
到了下午的时候,发现网站又打不开了,而且又远程连接不了服务器。进入阿里云控制台,查看监控发现cpu跑满了。只能再重启服务器,等重启完成后再远程连接上去,这次需要好好排查问题。
解决问题
当时首先想到的是中病毒了,先不管那么多,第一步是找到那些耗cpu的进程杀死。使用top命令,查看耗cpu的进程有哪些。一看就明白了,都是bzip2搞得鬼。
杀进程的过程发现一个问题,就是这些进程杀死了,过一会又出现了。这种现象,我知道肯定要找到他们的父进程,擒贼先擒王。
# ps -lA | grep bzip2 0 R 0 1965 1964 44 80 0 - 3435 - ? 00:01:43 bzip2 0 S 0 1981 1980 33 80 0 - 3435 pipe_w ? 00:00:56 bzip2 0 R 0 1997 1996 30 80 0 - 3435 - ? 00:00:31 bzip2 0 R 0 2013 2012 27 80 0 - 3435 - ? 00:00:07 bzip2 0 R 0 2024 2023 15 80 0 - 3435 - ? 00:00:00 bzip2
但是发现他们的ppid不是同一个,这就让我很疑惑了。我打算用进程树看看
pstree -up
这时候,我就知道了,原来是自己的定时脚本有问题。那么我需要做以下几件事:
关闭crond服务
crontab -e 将weekly.sh去掉
杀掉那些耗cpu的进程
# 关闭 [root@iz8vb626ci0aehwsivxaydz ~]# kill 1622 [root@iz8vb626ci0aehwsivxaydz ~]# systemctl status crond ● crond.service - Command Scheduler Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; vendor preset: enabled) Active: inactive (dead) since Tue 2019-11-12 10:44:32 CST; 10s ago Main PID: 1622 (code=exited, status=0/SUCCESS) # 修改crontab -e # 杀掉耗cpu进程,下面的命令执行了好几遍,才将所有耗cpu进程全部杀掉了 ps -lA | grep bzip2 | awk '{print $4}' | xargs -n 10 kill -9
问题原因与思考
刚开始,我以为是自己的shell脚本有问题,出现死循环导致问题出现。但是查看脚本,发现没有问题,没有死循环的情况出现。一时间,百思不得姐。
#!/bin/bash # 每周备份脚本 export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin export backdir=/backup/weekly # 备份目录 [ -z "$backdir" ] || mkdir -p $backdir dirs=(/etc /home /root /usr /var/spool/cron /var/spool/at) # 需要备份的目录 for dir in ${dirs[@]} do if [ ! -d $dir ];then continue fi cd $backdir tar -jcpf $(basename $dir)_$(date +%Y%m%d).tar.bz2 $dir done # 删除mtime大于30天的文件 find $backdir -mtime +30 -name *.tar.bz2 -exec rm -f {} \;
过了很长时间,终于找到了原因所在,原来是自己的定时任务写法有问题
* 3 * * 1 /root/bin/weekly.sh 1>/dev/null 2>&1
我原本的想法是每周1凌晨3点执行一次备份脚本,但是这样写的结果是每周一凌晨3点的每分钟都会执行该脚本一次。正确的写法应该如下:
# 每周一凌晨三点零一分执行该脚本 1 3 * * 1 /root/bin/weekly.sh 1>/dev/null 2>&1
问题解决了,原因也找到了。自己该写一个服务器资源监控脚本了。
The above is the detailed content of Record a server CPU full event. For more information, please follow other related articles on the PHP Chinese website!

Indispensable commands in Linux include: 1.ls: list directory contents; 2.cd: change working directory; 3.mkdir: create a new directory; 4.rm: delete file or directory; 5.cp: copy file or directory; 6.mv: move or rename file or directory. These commands help users manage files and systems efficiently by interacting with the kernel.

In Linux, file and directory management uses ls, cd, mkdir, rm, cp, mv commands, and permission management uses chmod, chown, and chgrp commands. 1. File and directory management commands such as ls-l list detailed information, mkdir-p recursively create directories. 2. Permission management commands such as chmod755file set file permissions, chownuserfile changes file owner, and chgrpgroupfile changes file group. These commands are based on file system structure and user and group systems, and operate and control through system calls and metadata.

MaintenanceModeinLinuxisaspecialbootenvironmentforcriticalsystemmaintenancetasks.Itallowsadministratorstoperformtaskslikeresettingpasswords,repairingfilesystems,andrecoveringfrombootfailuresinaminimalenvironment.ToenterMaintenanceMode,interrupttheboo

The core components of Linux include kernel, file system, shell, user and kernel space, device drivers, and performance optimization and best practices. 1) The kernel is the core of the system, managing hardware, memory and processes. 2) The file system organizes data and supports multiple types such as ext4, Btrfs and XFS. 3) Shell is the command center for users to interact with the system and supports scripting. 4) Separate user space from kernel space to ensure system stability. 5) The device driver connects the hardware to the operating system. 6) Performance optimization includes tuning system configuration and following best practices.

The five basic components of the Linux system are: 1. Kernel, 2. System library, 3. System utilities, 4. Graphical user interface, 5. Applications. The kernel manages hardware resources, the system library provides precompiled functions, system utilities are used for system management, the GUI provides visual interaction, and applications use these components to implement functions.

Linux maintenance mode can be entered through the GRUB menu. The specific steps are: 1) Select the kernel in the GRUB menu and press 'e' to edit, 2) Add 'single' or '1' at the end of the 'linux' line, 3) Press Ctrl X to start. Maintenance mode provides a secure environment for tasks such as system repair, password reset and system upgrade.

The steps to enter Linux recovery mode are: 1. Restart the system and press the specific key to enter the GRUB menu; 2. Select the option with (recoverymode); 3. Select the operation in the recovery mode menu, such as fsck or root. Recovery mode allows you to start the system in single-user mode, perform file system checks and repairs, edit configuration files, and other operations to help solve system problems.

The core components of Linux include the kernel, file system, shell and common tools. 1. The kernel manages hardware resources and provides basic services. 2. The file system organizes and stores data. 3. Shell is the interface for users to interact with the system. 4. Common tools help complete daily tasks.


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

Atom editor mac version download
The most popular open source editor

MinGW - Minimalist GNU for Windows
This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

Zend Studio 13.0.1
Powerful PHP integrated development environment

WebStorm Mac version
Useful JavaScript development tools
