How to solve nginx+php-fpm service HTTP status code 502

For one of our web projects, due to the increase in new cities, the number of visits has increased and the pressure on the db has increased. As a business party that provides interfaces, a large number of "502" requests have been reported recently by downstream feedback.

502, bad gateway, is usually an error in upstream (here is PHP). For PHP, the common cause of 502 is that the script execution exceeds the timeout setting time, or the timeout setting is too large, causing the PHP process to take a long time It cannot be released and there are no idle worker processes to pick up guests.

Our project is caused by the PHP execution time setting being too short. In this case, you can first increase the PHP execution time appropriately and ensure that 502 is cleared first. Optimization will take more time after all.

There are two options to control php execution time, max_execution_time in php.ini and request_terminate_timeout in php-fpm. request_terminate_timeout can override max_execution_time, so if you don’t want to change the global php.ini, just change php- The configuration of fpm is enough.

Next I will analyze in detail why the execution of the php script exceeds the set time and causes nginx to return 502.

Let’s set the scene first and let the problem reoccur:

nginx and php only start one worker each for easy tracking.

php-fpm's request_terminate_timeout is set to 3s.

Test script test.php

sleep(20);
echo 'ok';

go go go:

Visit www.v.com/test.php in the browser, and 404 will appear as expected after 3 seconds. ? ? ? what? ? ?

It’s a bad start, quickly take a look at the nginx configuration file

This location configuration is when a 5xx error occurs Jump to a nice-looking interface, but I don’t have the file 50x.html under /usr/share/nginx/html. So I got a 404. Doesn't this affect the accuracy of my judgment of the problem? Just comment it out! Visit again, wait 3 seconds, and finally the 'normal' interface comes out.

The environment is good, let’s follow the routine below. Follow the troubleshooting routine for web problems. Let’s take a look at the error log first:

nginx:

The errors reported are recv() failed (104: connection reset by peer.

recv failed and the connection was reset. Why was the connection reset? Set it? Isn’t it consistent?

We are looking at the error log of php-fpm:

(Note that the php_admin_value[error_log] option in php-fpm specifies the error log of php, which will be overwritten in php.ini. But here we are not looking at php errors, but at php-fpm errors. The error log of php-fpm is specified by the error_log option in php-fpm.conf.)

Each request generates 2 warnings and 1 notice:

warning: The script execution timed out and terminated.

warning: The child process received sigterm The signal exited.

Notice: A new child process was started (because I set pm.min_spare_servers = 1)

It seems that if the worker process of php times out, it will not only terminate the script execution , and the worker process will also exit. It seems that the nginx error connection is reset because the php worker process exits (in the tcp connection, if one party is disconnected, it will send rst to the other party)

Through the log We can already know that the execution of the php script times out and the worker sub-process exits, causing nginx to report an error connection reset by peer. Let’s use strace to see the situation of php and nginx:

php：

1.Accept an nginx connection request (socket, bind, and listen are all completed in the master). You can see that the port of nginx is 47039, and the data is read from fd0, which is from the standard input. This is stipulated by the fast-cgi protocol. The connected descriptor after accept is 3.

2. Read the data passed by nginx from fd3, in the fastcgi protocol format, and received 856 bytes. Why read5 What about times?

Because the fastcgi protocol data packet is 8-byte aligned and consists of a packet header and a packet body. And it will first send a request packet, including some request id, version, type and other information (the header and body each occupy 8 bytes), and then send a params packet to pass the get parameters and environment variables (the header is 8 bytes) , the packet body becomes longer), and finally a params data packet without a packet body and only a packet header is sent, indicating the end of parameter sending (8 bytes of packet header). So the first three reads are used to read the header and body of the request packet, as well as the header of the params packet. The fourth read is to read the real data, and the last read is to read the header of the last params packet. Therefore, the data transmitted by nginx should be 8 8 8 856 8 = 896 bytes (which can correspond to the transmission bytes of nginx below). Note that if it is post mode, stdin data packets will also be sent.

3. Set sleep for 20s, which is sleep(20) in the php program. After that, because the process is terminated, there will be no more. The strace program also exited.

nginx:

##1.Accept the request to the browser. You can see that the port on the browser side is 56434, the ip is 192.168.1.105, and it has been established. The connected fd is 3.

2. Receive data from fd3, http protocol.

3. Create a socket, fd21, to establish a connection with php.

4. Connect to fd21, you can see that the connection is the 9000 port of the local machine. Here nginx and php-fpm use ip socket connection method. If nginx and php-fpm are deployed on one machine, unix can be considered. domain socket.

5. Write data to fd21 in fast-cgi protocol format. We see that the written length is 896, which corresponds to the length received by PHP above.

6. The recvfrom function returns econnreset (connection reset by peer) from fd21

7. Write error information to fd9. It can be inferred that fd9 is the file descriptor of the nginx error log.

8. Close the connection with fd21.

9. Write 502 bad gateway to fd3, which is the information returned to the browser.

10. Write an access log to fd8. It can be inferred that fd8 is the file descriptor of the nginx access log.

Let’s verify the inference of nginx access log and error log. You can see that it is indeed fd8, fd9, and in write mode.

Then we might as well take a look at the transmission of the entire network packet during this process:

Capture packets through tcpdump, it is more convenient to use an artifact to view it.

Because I only want to see the communication between nginx and php, and I know that the port of nginx is 47039, I can filter out the corresponding package through tcp.srcport==47039.

You can see the process of data interaction between nginx and php-fpm: 47039->9000 establishes a three-way handshake, then sends data to 9000, 9000 replies with ack, and 9000 after 3 seconds Reply to rst. Nothing wrong.

Note:

syn, fin each occupy a sequence number

ack, rst does not occupy a sequence number (the reqnum and acknum of the two packages 28 and 29 are the same )

The sequence number is plus 1 for each byte (896 bytes are sent in 29 packets, while the seq of 29 packets is 4219146879, and the ack of 30 packets is 4219147775, which is exactly a difference of 896)

rst No Reply required.

The above is the detailed content of How to solve nginx+php-fpm service HTTP status code 502. For more information, please follow other related articles on the PHP Chinese website!