nginx signal set example analysis

Scenario Reproduction

Below I will use a native nginx to reproduce this process on my virtual machine with fedora26 installed. The nginx version I use is the latest 1.13.4

First start nginx

You can see that both master and worker are already running.

Then we send a sigusr2 signal to the master. When the nginx core receives this signal, a hot update will be triggered.

You can see that the new master and the workers forked by the master are already running. At this time, we then send a sigwinch signal to the old master, and the old master receives this After the signal, sigquit will be sent to its worker, so the worker process of the old master will exit:

At this time, only the old master, the new master and the new master are left. The master's worker is running, which is similar to the online operation at that time.

Then we use the stop command:

We will find that the new master and its workers have exited, while the old master is still running. And generated workers. This was the situation online at that time.

In fact, this phenomenon is related to the design of nginx itself: when the old master is ready to fork a new master, it will rename the nginx.pid file to nginx.pid.oldbin, and then rename it to nginx.pid.oldbin. The new master from the fork will create a new nginx.pid. This file will record the pid of the new master. nginx believes that after the hot update is completed, the mission of the old master is almost over, and it will exit at any time, so all subsequent operations should be taken over by the new master. Of course, it is invalid to attempt another hot update by sending sigusr2 to the new master without the old master exiting. The new master will just ignore this signal and continue its own work.

Problem Analysis

What’s even more unfortunate is that the lua table we mentioned above, the lua file that defines it, has been loaded into memory by luajit as early as when the init_by_lua hook is run. And compiled into bytecode, then obviously the old master must not have this lua table, because the part of lua code it loads is an old version.

The Lua code that indexes the table is not used during init_by_lua. These codes are loaded in the worker process. At this time, the codes in the project directory are all the latest, so the worker process The latest code is loaded. If these worker processes process relevant requests, a Lua runtime error will occur, and the external manifestation will be the corresponding http 500.

After absorbing this lesson, we need to shut down our nginx service more rationally. Therefore, a more reasonable nginx service startup and shutdown script is necessary. Some scripts circulated on the Internet do not deal with this phenomenon. We should refer to the official script provided by nginx.

This code is quoted from nginx official /etc/init.d/nginx.

nginx signal set

Next, let’s comprehensively sort out the nginx signal set. The source code details will not be involved here. Interested students can read the relevant source code by themselves.

We have two ways to send signals to the master process, one is to operate through nginx -s signal, and the other is to send it manually through the kill command.

The principle of the first method is to generate a new process, which gets the pid of the master process through the nginx.pid file, then sends the corresponding signal to the master, and then exits. This process is called signaller.

The second method requires us to understand the mapping of nginx -s signal to real signals. The following table is their mapping relationship:

operation signal
reload sighup
reopen sigusr1
stop sigterm
quit sigquit
hot update sigusr2 & sigwinch & sigquit
stop vs quit

#stop sends the sigterm signal to indicate a forced exit, and quit sends sigquit to indicate a graceful exit. The specific difference is that after the worker process receives the sigquit message (note that the signal is not sent directly, so the message is used instead), it will close the listening socket, close the current idle connection (the connection that can be preempted), and then process it in advance All timer events exit at the end. Unless there are special circumstances, quit should be used instead of stop.

reload

After the master process receives the sighup, it will re-parse the configuration file, apply for shared memory, and a series of other tasks, and then generate a batch of new worker processes, and finally send them to the old ones. The worker process sends the message corresponding to sigquit, and finally the restart operation is seamlessly realized.

reopen

After the master process receives sigusr1, it will reopen all open files (such as logs), and then send sigusr1 information to each worker process. After the worker process receives the signal, it will perform the same operation. Reopen can be used for log cutting. For example, nginx officially provides a solution:

sleep 1 is required here because the master process sends the sigusr1 message to the worker process. There is a window of time between actually reopening access.log. At this time, the worker process still writes logs to the file access.log.0. By sleeping 1s, the integrity of the access.log.0 log information is guaranteed (if compression is performed directly without sleep, log loss is likely to occur).

hot update

Sometimes we need to perform binary hot update. nginx includes this function when designing, but it cannot be completed through the command line provided by nginx. We need to send it manually Signal.

Through the recurrence of the above problem, you should have understood how to perform hot update. We first need to send sigusr2 to the current master process, and then the master will rename nginx.pid to nginx.pid.oldbin. Then fork a new process. The new process will use the execve system call to replace the current process image with the new nginx elf file and become the new master process. After the new master process starts up, it will perform configuration file parsing and other operations, and then fork out the new worker process to start working.

Then we send the sigwinch signal to the old master, and then the old master process will send the sigquit message to its worker process, causing the worker process to exit. Sending sigwinch and sigquit to the master process will cause the worker process to exit, but the former will not cause the master process to exit as well.

Finally, if we feel that the old master process has completed its mission, we can send it the sigquit signal to let it exit.

How the worker process handles the signal message from the master

In fact, the master process communicates with the worker process, not using the kill function, but using the nginx channel implemented through the pipe, the master process Write information (such as signal information) to one end of the pipe, and the worker process receives information from the other end. The nginx channel event is added to the event scheduler (such as epoll, kqueue) when the worker process just starts, so when there is When data is sent from the master, it can be notified by the event scheduler.

nginx is designed this way for a reason. As an excellent reverse proxy server, nginx pursues extreme high performance, and the signal handler will interrupt the running of the worker process, causing all events to be suspended. time window, which has a certain loss on performance.

Many people may think that when the master process sends information to the worker process, the worker process will immediately respond with corresponding operations. However, the worker process is very busy. It is constantly processing network events and timer events. After calling the nginx channel event handler, nginx only processes some flags. These actions are actually executed after a round of event scheduling is completed. Therefore, there is a time window in between. Especially when the business is complex and the traffic is huge, this window may be enlarged. This is why the log cutting plan officially provided by nginx requires sleep 1s.

Of course, we can also bypass the master process and send signals directly to the worker process. The signals that the worker can handle are

signal effect
sigint force exit
sigterm force exit
sigquit Graceful exit
sigusr1 Reopen the file

The above is the detailed content of nginx signal set example analysis. For more information, please follow other related articles on the PHP Chinese website!