When traffic comes, time slows down. Is it because there are not enough PHP-FPM processes?

感觉PHP-FPM进程数不够？

作为一个 phper，用的最多的架构就是 LNMP。每次一到流量来了，我们的服务就从原来的 几百毫秒到几秒的时间。这个时候我们各种猜测，mysql 有慢 sql，redis 有大 key，php-fpm 进程数不够等等情况。其中可以通过业务的一些日志来排查如上情况。我们这次主要证明的却是 php-fpm 进程数不够情况的实践。

重现现场

1.将我本地的的 PHP-FPM 进程数调整为 2

#vim /etc/php-fpm.d/www.conf
pm = static
pm.max_children = 2

2.使用 ab 来压测接口

$ ab -c 40  -n 3000 http://127.0.0.1/group/check_groups
Server Software:        nginx/1.16.0
Server Hostname:        miner_platform.cn
Server Port:            80
Document Path:          /group/check_groups
Document Length:        44 bytes
Concurrency Level:      40
Time taken for tests:   29.384 seconds
Complete requests:      3000
Failed requests:        0
Write errors:           0
Total transferred:      699000 bytes
HTML transferred:       132000 bytes
Requests per second:    102.10 [#/sec] (mean)
Time per request:       391.788 [ms] (mean)
Time per request:       9.795 [ms] (mean, across all concurrent requests)
Transfer rate:          23.23 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:   306  344  80.6    318    3558
Waiting:      306  343  80.5    318    3555
Total:        307  344  80.6    318    3558
Percentage of the requests served within a certain time (ms)
  50%    318
  66%    322
  75%    333
  80%    369
  90%    428
  95%    461
  98%    508
  99%    553
 100%   3558 (longest request)

尝试解决问题

1. PHP-FPM STATUS

我们发现接口 318ms 到 3.558s 的都有，那我们如何知道 php-fpm 进程少不够导致这个问题呢？换一种说话有什么办法能让我们知道 php-fpm 内部是处理不过来吗？ 这个时候我们就需要打开 php-fpm 内置 status 了。

详细步骤参考：https://www.php.cn/php-weizijiaocheng-485633.html

$ curl http://127.0.0.1/status.php
pool:                 www
process manager:      static
start time:           29/Nov/2021:18:27:38 +0800
start since:          6493
accepted conn:        3136
listen queue:         38
max listen queue:     39
listen queue len:     128
idle processes:       0
active processes:     2
total processes:      2
max active processes: 2
max children reached: 0
slow requests:        0

具体详细的字段可以参见上面的链接，有详细说明，我们主要说下几个参数

• listen queue：这个就是此时此刻我们的 php-fpm 作为服务端，处于 accept 队列 的数量。

• max listen queue： 从 php-fpm 进程启动到现在处于等待连接的最大数量（说白了，就是我们上面说的 listen queue 的最大值持久化）

• listen queue len ： 有过 socket 网络编程经验的同学都知道。int listen(int sockfd, int backlog); 是可以设置该参数，但是他和系统设置有关系。

2. netstat 查看链接状态

我们得到的结论是：当 php-fpm 进程处理不过来的时候，请求就会放在 accept 队列，知道了这个情况以后，我们甚至不需要通过 status。

第一行表示的监听 socket， Recv-Q 表示 accept queue 长度。

$netstat -antp | grep php-fpm
tcp       38      0 127.0.0.1:9000          0.0.0.0:*               LISTEN      97/php-fpm: master  
tcp        8      0 127.0.0.1:9000          127.0.0.1:55540         ESTABLISHED 964/php-fpm: pool w 
tcp        8      0 127.0.0.1:9000          127.0.0.1:55536         ESTABLISHED 965/php-fpm: pool w

综上我们知道了，当 PHP-FPM 进程数不够的时候，nginx 客户端请求的连接的 accept 队列 长度就会变大。这样就完了吗？不，我们还需要去分析为什么能得到这个现象。

原理分析

简述 PHP-FPM 工作过程

首先我们需要简单里说一说 php-fpm 的工作过程。我们就简单模型一下它的伪代码（这里只为了表述整个 socket 的过程）

// 1. 创建 socket
$socket = socket_create(AF_INET, SOCK_STREAM, 0);
// 2. 绑定socket
socket_bind($socket, "0.0.0.0", 9000);
// 3. 监听 socket
socket_listen($socket, 5);
for($i=0;$i<2;$i++) {
    $pid = pcntl_fork()
    // 4. 创建2个进程
    if ($pid == 0) {
        // 5. 子进程接受socket
        while($fd = socket_accept($socket)) {
            echo "客户端${fd}连接" . PHP_EOL;
            $tmp = socket_read($fd, 1024);
            echo "client data:" . $tmp . PHP_EOL;
            $data = "HTTP/1.1 200 ok\r\nContent-Length:2\r\n\r\nhi";
            socket_write($fd, $data, strlen($data));
        }    
        exit;
    }
}
// 5. 监听子进程退出
// 其他 TODO

1.master 进程创建了监听 socket，但是不处理业务正在

2.work 进程接受同步堵塞接受请求（堵塞在 accept），然后处理业务。

抓取 nginx->php-fpm socket

我们知道了 php-fpm 大概工作的过程，这个时候我们就需要通过一次请求大概知道 nginx 与 php-fpm 交互的过程。

$curl http://miner_platform.cn/group/check_groups
{"code":10006,"message":"sign\u65e0\u6548."}

1.nginx 系统调用

需要关注的点都在这个里面注释了。抓取的是 nginx work 进程

$ strace -f -s 64400 -p 958
 strace: Process 958 attached
 epoll_wait(8, [{EPOLLIN, {u32=1226150064, u64=94773974503600}}], 512, -1) = 1
 accept4(6, {sa_family=AF_INET, sin_port=htons(46616), sin_addr=inet_addr("127.0.0.1")}, [112->16], SOCK_NONBLOCK) = 3
 epoll_ctl(8, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=1226159737, u64=94773974513273}}) = 0
 epoll_wait(8, [{EPOLLIN, {u32=1226159737, u64=94773974513273}}], 512, 60000) = 1
 recvfrom(3, "GET /group/check_groups HTTP/1.1\r\nUser-Agent: curl/7.29.0\r\nHost: miner_platform.cn\r\nAccept: */*\r\n\r\n", 1024, 0, NULL, NULL) = 99
 stat("/data/miner_platform/src/public/group/check_groups", 0x7ffcb593d1b0) = -1 ENOENT (No such file or directory)
 stat("/data/miner_platform/src/public/group/check_groups", 0x7ffcb593d1b0) = -1 ENOENT (No such file or directory)
 epoll_ctl(8, EPOLL_CTL_MOD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=1226159737, u64=94773974513273}}) = 0
 lstat("/data", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
 lstat("/data/miner_platform", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
 lstat("/data/miner_platform/src", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
 lstat("/data/miner_platform/src/public", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
 getsockname(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("127.0.0.1")}, [112->16]) = 0
 // 1. 创建 socket    
 socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 11
 ioctl(11, FIONBIO, [1])                 = 0
 epoll_ctl(8, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=1226163953, u64=94773974517489}}) = 0
 // 2. 连接 127.0.0.1:9000    
 connect(11, {sa_family=AF_INET, sin_port=htons(9000), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)    
 epoll_wait(8, [{EPOLLOUT, {u32=1226159737, u64=94773974513273}}, {EPOLLOUT, {u32=1226163953, u64=94773974517489}}], 512, 60000) = 2
 getsockopt(11, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
 // 3. 按照FASTCGI协议写入这次请求     
 writev(11, [{iov_base="\1\1\0\1\0\10\0\0\0\1\0\0\0\0\0\0\1\4\0\1\2!\7\0\17)SCRIPT_FILENAME/data/miner_platform/src/public/index.php\f\0QUERY_STRING\16\3REQUEST_METHODGET\f\0CONTENT_TYPE\16\0CONTENT_LENGTH\v\nSCRIPT_NAME/index.php\v\23REQUEST_URI/group/check_groups\f\nDOCUMENT_URI/index.php\r\37DOCUMENT_ROOT/data/miner_platform/src/public\17\10SERVER_PROTOCOLHTTP/1.1\16\4REQUEST_SCHEMEhttp\21\7GATEWAY_INTERFACECGI/1.1\17\fSERVER_SOFTWAREnginx/1.16.0\v\tREMOTE_ADDR127.0.0.1\v\5REMOTE_PORT46616\v\tSERVER_ADDR127.0.0.1\v\2SERVER_PORT80\v\21SERVER_NAMEminer_platform.cn\17\3REDIRECT_STATUS200\17\vHTTP_USER_AGENTcurl/7.29.0\t\21HTTP_HOSTminer_platform.cn\v\3HTTP_ACCEPT*/*\0\0\0\0\0\0\0\1\4\0\1\0\0\0\0\1\5\0\1\0\0\0\0", iov_len=592}], 1) = 592
 epoll_wait(8, [{EPOLLIN|EPOLLOUT, {u32=1226163953, u64=94773974517489}}], 512, 60000) = 1
 // 4. 接受 PHP-FPM响应结果   
 recvfrom(11, "\1\6\0\1\0\257\1\0X-Powered-By: PHP/7.2.16\r\nCache-Control: no-cache, private\r\nDate: Wed, 01 Dec 2021 12:24:52 GMT\r\nContent-Type: application/json\r\n\r\n{\"code\":10006,\"message\":\"sign\\u65e0\\u6548.\"}\0\1\3\0\1\0\10\0\0\0\0\0\0\0\"}\0", 4096, 0, NULL, NULL) = 200
 epoll_wait(8, [{EPOLLIN|EPOLLOUT|EPOLLRDHUP, {u32=1226163953, u64=94773974517489}}], 512, 60000) = 1
 readv(11, [{iov_base="", iov_len=3896}], 1) = 0
 // 5. 关闭这次socket连接    
 close(11)                               = 0
 // 6. 响应给浏览器    
 writev(3, [{iov_base="HTTP/1.1 200 OK\r\nServer: nginx/1.16.0\r\nContent-Type: application/json\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nX-Powered-By: PHP/7.2.16\r\nCache-Control: no-cache, private\r\nDate: Wed, 01 Dec 2021 12:24:52 GMT\r\n\r\n", iov_len=222}, {iov_base="2c\r\n", iov_len=4}, {iov_base="{\"code\":10006,\"message\":\"sign\\u65e0\\u6548.\"}", iov_len=44}, {iov_base="\r\n", iov_len=2}, {iov_base="0\r\n\r\n", iov_len=5}], 5) = 277
 write(5, "127.0.0.1 - - [01/Dec/2021:20:24:52 +0800] \"GET /group/check_groups HTTP/1.1\" 200 55 \"-\" \"curl/7.29.0\" \"-\" 1.029 127.0.0.1:9000 200 1.030\n", 138) = 138
 setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
 epoll_wait(8, [{EPOLLIN|EPOLLOUT|EPOLLRDHUP, {u32=1226159737, u64=94773974513273}}], 512, 65000) = 1
 recvfrom(3, "", 1024, 0, NULL, NULL)    = 0
 close(3)                                = 0
 epoll_wait(8,

2.php-fpm 系统调用

抓取了 php-fpm work 进程

// 1. accept 接收到了 nginx(127.0.0.1:45512 ) 客户端发送的数据
965   accept(9, {sa_family=AF_INET, sin_port=htons(45512), sin_addr=inet_addr("127.0.0.1")}, [112->16]) = 4
中间省略了许多
// 2. 响应给客户端
965   write(4, "\1\6\0\1\0\257\1\0X-Powered-By: PHP/7.2.16\r\nCache-Control: no-cache, private\r\nDate: Wed, 01 Dec 2021 12:37:18 GMT\r\nContent-Type: application/json\r\n\r\n{\"code\":10006,\"message\":\"sign\\u65e0\\u6548.\"}\0\1\3\0\1\0\10\0\0\0\0\0\0\0p\0\0", 200) = 200
// 3. 不给给这个socket 写数据了
965   shutdown(4, SHUT_WR)              = 0
// 4. 接受nginx(127.0.0.1:45512 )客户端数据 
965   recvfrom(4, "\1\5\0\1\0\0\0\0", 8, 0, NULL, NULL) = 8
// 5. 接受nginx(127.0.0.1:45512 )客户端数据 
965   recvfrom(4, "", 8, 0, NULL, NULL) = 0
// 6. 关闭这个连接
965   close(4)                          = 0
965   lstat("/data/miner_platform/src/vendor/composer/../../app/Http/Middleware/BusinessHeaderCheck.php", {st_mode=S_IFREG|0777, st_size=989, ...}) = 0
965   stat("/data/miner_platform/src/app/Http/Middleware/BusinessHeaderCheck.php", {st_mode=S_IFREG|0777, st_size=989, ...}) = 0
965   chdir("/")                        = 0
965   times({tms_utime=3583, tms_stime=1977, tms_cutime=0, tms_cstime=0}) = 4315309933
965   setitimer(ITIMER_PROF, {it_interval={tv_sec=0, tv_usec=0}, it_value={tv_sec=0, tv_usec=0}}, NULL) = 0
965   fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
965   setitimer(ITIMER_PROF, {it_interval={tv_sec=0, tv_usec=0}, it_value={tv_sec=0, tv_usec=0}}, NULL) = 0
965   accept(9,

TCP 三次握手

上面我们已经清楚了一次请求，请求并发高的时候流程也是如此，这个时候我们就引出了下面这个图与我们上面描述的过程是一样的，只是细化了三次握手的过程。这个时候我们引出了 sync queue 和 accept queue。

• 我们调用 listen (上面是 php-fpm master 进程执行的)，于此同时内核创建了两个队列 sync queue 和 accept queue

• 三次握手第二步当 Server (指的是 php-fpm master 进程) 发送了 SYN+ACK 报文后，此时会将这个信息放入到 sync queue

• 当三次握手完成时，未被应用 (指的是 php-fpm work 进程) 调用 accept 取走的连接队列。此时的 socket 处于 ESTABLISHED 状态。每次应用调用 accept() 函数会移除队列头的连接。如果队列为空，accept () 通常会阻塞。全连接队列也被称为 accept queue。

结论

After the above analysis, we know what sync queue and accept queue are. The application, accept queue, and kernel are a production and consumption model. The kernel is the producer, accept queue stores queue information, and the application is the consumer. Students who have used queues know that when concurrency is high, there will be more data in the queue, or the slow consumption of producers will cause subsequent connection processing to become slower and slower. Therefore, the usual approach is to add consumers to improve Consumption speed of these two options. This also coincides with our phenomenon above.

The above is the detailed content of When traffic comes, time slows down. Is it because there are not enough PHP-FPM processes?. For more information, please follow other related articles on the PHP Chinese website!