How to analyze gunicorn Arbiter source code

As mentioned earlier, Arbiter is the core of the gunicorn master process. Arbiter is mainly responsible for managing worker processes, including starting, monitoring, and killing worker processes. At the same time, Arbiter can also hot update (reload) App applications or upgrade gunicorn online when certain signals occur. The core code of Arbiter is in one file, and the amount of code is not large. The source code is here: https://github.com/benoitc/gunicorn.

Arbiter mainly has the following methods:

setup:

Processing configuration items, the most important ones are the number of workers and the worker working model

init_signal：

Register signal processing function

handle_xxx:

Specific processing function for each signal

kill_worker, kill_workers:

Send a signal to the worker process

spawn_worker, spawn_workers:

Fork to create a new worker Process

murder_workers:

Kill worker processes that have not responded for a period of time

manage_workers:

Based on the number of workers in the configuration file and the number of currently active workers, decide whether to fork or kill the worker process

reexec:

After receiving the signal SIGUSR2 call, online Upgrade gunicorn

#reload:

After receiving the signal SIGHUP call, the worker process will be started based on the new configuration and the previous worker process will be killed

sleep：

When there is no signal processing, use the select timeout to sleep and can be woken up

wakeup：

Wake up the process by writing messages to the pipe

run:

Main loop

The only functions of Arbiter that are actually called by other codes (Application) __init__ and run methods, in one line of code:

Arbiter(self).run()

The self in the above code is the Application instance, where_ _init__ calls setup to set configuration items. The following is the pseudo code of the run method

def run()
    self.init_signal()
    self.LISTENERS = create_sockets(self.cfg, self.log)
    self.manage_workers()    while True:        if no signal in SIG_QUEUE
            self.sleep()        else:
            handle_signal()

## About fork child process

fork child process The code is in spawn_worker, the source code is as follows:

Arbiter.spawn_worker

Main process:

(1) Load worker_class and instantiate (default is synchronous model SyncWorker)

(2) The parent process (master process) returns after forking, and all subsequent logic is run in the child process

(3) Call worker.init_process to enter the loop, and all the new channel IELTS training The work is all in this loop

(4) After the loop ends, call sys.exit(0)

(5) Finally, in finally, record the exit of the worker process

The following is a bit of code I wrote myself, which simplifies the main fork process

 1 # prefork.py 2 import sys 3 import socket 4 import select 5 import os 6 import time 7   8 def do_sub_process(): 9     pid = os.fork()10     if pid < 0:11         print &#39;fork error&#39;12         sys.exit(-1)13     elif pid > 0:14         print 'fork sub process %d'  % pid15         return16  17     # must be child process18     time.sleep(1)19     print 'sub process will exit', os.getpid(), os.getppid()20     sys.exit(0)21  22 def main():23     sub_num = 224     for i in range(sub_num):25         do_sub_process()26     time.sleep(10)27     print 'main process will exit', os.getpid()28  29 if __name__ == '__main__':30     main()

Output in the test environment:

fork sub process 9601

fork sub process 9602

sub process will exit 9601 9600

sub process will exit 9602 9600

Main process will exit 9600

It should be noted that line 20 calls

sys.exit to ensure the end of the child process, otherwise the for loop in the main function will continue. , and the subsequent logic. Comment out line 19 and run again, and you will understand by looking at the output.

About killing child processes

It is very simple for the master process to kill the worker process. It sends a signal directly. The source code is as follows:

 1     def kill_worker(self, pid, sig): 2         """\ 3         Kill a worker 4  5         :attr pid: int, worker pid 6         :attr sig: `signal.SIG*` value 7          """ 8         try: 9             os.kill(pid, sig)10         except OSError as e:11             if e.errno == errno.ESRCH:12                 try:13                     worker = self.WORKERS.pop(pid)14                     worker.tmp.close()15                     self.cfg.worker_exit(self, worker)16                     return17                 except (KeyError, OSError):18                     return19             raise

About sleep and wakeup

Let’s take a look at Arbiter’s sleep and wakeup. Arbiter will "sleep" when there is no signal to process. Of course, it does not actually call time.sleep, otherwise the signal will not be processed immediately when it comes. The implementation here is more clever, using pipes and select timeout. Just look at the code and you will know

        def sleep(self):        """\
        Sleep until PIPE is readable or we timeout.
        A readable PIPE means a signal occurred.        """
            ready = select.select([self.PIPE[0]], [], [], 1.0) # self.PIPE = os.pipe()
            if not ready[0]: 
                return
            while os.read(self.PIPE[0], 1):                pass

The comments in the code are very clear, either PIPE is readable and returns immediately, or the wait times out. The pipe is readable because a signal occurs. Here's a look at the pipe function

• os.pipe()

• Create a pipe. Return a pair of file descriptors

  (r,w) usable for reading and writing, respectively.

Then let’s take a look at when the pipe is readable: it must be to What is written to the pipe, this is the function of the wakeup function

        def wakeup(self):            """
            Wake up the arbiter by writing to the PIPE            """
            os.write(self.PIPE[1], b'.')

Finally, the signal processing of Arbiter is attached:

Exit, INT: Quick shutdown

TERM: Graceful shutdown. Waits for the worker to complete its current request until it times out.

HUP: Reload configuration, start new worker processes with new configuration, and shut down old worker processes gracefully. Gunicorn will also load the new version if the application is not preloaded (using the --preload option).

TTIN: Increase the number of processes by one

TTOU: Decrease the number of processes by one

USR1: Reopen the log file

USR2: Upgrade on the fly Gunicorn. A separate term signal should be used to terminate the old process. This signal can also be used to use preloaded new versions of the application.

Winch: Gracefully shut down worker processes when Gunicorn is daemonized.

The above is the detailed content of How to analyze gunicorn Arbiter source code. For more information, please follow other related articles on the PHP Chinese website!