In-depth analysis of the select module in python

Introduction

The select module in Python focuses on I/O multiplexing and provides three methods: select poll and epoll (the latter two are available in Linux, and windows only supports select). In addition The kqueue method (freeBSD system) is also provided.

select method

The process specifies which file descriptors the kernel listens for (up to 1024 fds) and which events, when no file descriptor events occur. , the process is blocked; when one or more file descriptor events occur, the process is awakened.

When we call select():

1. Context switch to kernel state

2. Copy fd from user space to kernel space

3. The kernel traverses all fds to see if the corresponding event occurs

4. If it does not occur, the process is blocked. When the device driver generates an interrupt or timeout occurs, the process is awakened and traversed again

　5. Return the traversed fd

　6. Copy fd from kernel space to user space

fd:file descriptor file descriptor

fd_r_list, fd_w_list, fd_e_list = select.select(rlist, wlist, xlist, [timeout])

Parameters: Accepts four parameters (the first three are required)

rlist: wait until ready for reading

wlist: wait until ready for writing

xlist: wait for an “exceptional condition”

timeout: timeout time

Return value: three lists

The select method is used to monitor file descriptors (when the file descriptor conditions are not met, select will block). When the status of a file descriptor changes, three lists will be returned

1. When the fd in the sequence of parameter 1 meets the "readable" condition, the changed fd is obtained and added to fd_r_list

2. When the sequence of parameter 2 contains fd, all the fd in the sequence are fd is added to fd_w_list

3. When an error occurs in the fd in the sequence of parameter 3, the fd with the error will be added to fd_e_list

4. When the timeout is empty, then select will block until the monitored handle changes

When the timeout = n (positive integer), then if there is no change in the monitored handle, select will block for n seconds, and then return three nulls list, if the monitored handle changes, it will be executed directly.

Example:

Use select to implement a concurrent server

import socket
import select
 
s = socket.socket()
s.bind(('127.0.0.1',8888))
s.listen(5)
r_list = [s,]
num = 0
while True:
 rl, wl, error = select.select(r_list,[],[],10)
 num+=1
 print('counts is %s'%num)
 print("rl's length is %s"%len(rl))
 for fd in rl:
  if fd == s:
   conn, addr = fd.accept()
   r_list.append(conn)
   msg = conn.recv(200)
   conn.sendall(('first----%s'%conn.fileno()).encode())
  else:
   try:
    msg = fd.recv(200)
    fd.sendall('second'.encode())
   except ConnectionAbortedError:
    r_list.remove(fd)
 
s.close()

import socket
 
flag = 1
s = socket.socket()
s.connect(('127.0.0.1',8888))
while flag:
 input_msg = input('input>>>')
 if input_msg == '0':
  break
 s.sendall(input_msg.encode())
 msg = s.recv(1024)
 print(msg.decode())
 
s.close()

On the server side we can see that we need to keep calling select, which means ：

　1 When there are too many file descriptors, copying the file descriptors between user space and kernel space will be very time-consuming

　2 When there are too many file descriptors, the kernel traverses the file descriptors It is also a waste of time

　3 Select only supports a maximum of 1024 file descriptors

poll is not much different from select and will not be introduced in this article

epoll method:

epoll has improved select very well:

　1. The solution of epoll is in the epoll_ctl function. Each time a new event is registered to the epoll handle, all fds will be copied into the kernel instead of repeated copies during epoll_wait. epoll ensures that each fd will only be copied once during the entire process.

　2. epoll will traverse the specified fd during epoll_ctl (this time is essential) and specify a callback function for each fd. When the device is ready and wakes up the waiters on the waiting queue, This callback function will be called, and this callback function will add the ready fd to a ready linked list. The job of epoll_wait is actually to check whether there is a ready fd in this ready list

　3. epoll has no additional restrictions on file descriptors

select.epoll(sizehint=-1, flags= 0) Create an epoll object

epoll.close()

Close the control file descriptor of the epoll object.Close the file descriptor of the epoll object

epoll.closed

True if the epoll object is closed. Check whether the epoll object is closed

epoll.fileno()

Return the file descriptor number of the control fd. Return the file descriptor of the epoll object

epoll.fromfd(fd)

Create an epoll object from a given file descriptor. Create an epoll object based on the specified fd

epoll.register(fd[, eventmask])

Register a fd descriptor with the epoll object. To the epoll object Register fd and the corresponding event in

epoll.modify(fd, eventmask)

Modify a registered file descriptor. Modify the event of fd

epoll.unregister(fd)

Remove a registered file descriptor from the epoll object.Unregister

epoll.poll(timeout=-1, maxevents=-1)

Wait for events. timeout in seconds (float) blocks until the registered fd event occurs, and a dict will be returned with the format: {(fd1,event1),(fd2,event2) ,...(fdn,eventn)}

Event:

EPOLLIN Available for read The readable status symbol is 1

EPOLLOUT Available for write The writable status symbol is 4

EPOLLPRI Urgent data for read

EPOLLERR Error condition happened on the assoc. fd The error status symbol is 8

EPOLLHUP Hang up happened on the assoc. fd Hang status

EPOLLET Set Edge Trigger behavior, the default is Level Trigger behavior 默认为水平触发，设置该事件后则边缘触发

EPOLLONESHOT Set one-shot behavior. After one event is pulled out, the fd is internally disabled

EPOLLRDNORM Equivalent to EPOLLIN

EPOLLRDBAND Priority data band can be read.

EPOLLWRNORM Equivalent to EPOLLOUT

EPOLLWRBAND Priority data may be written.

EPOLLMSG Ignored.

水平触发和边缘触发：

Level_triggered(水平触发，有时也称条件触发)：当被监控的文件描述符上有可读写事件发生时，epoll.poll()会通知处理程序去读写。如果这次没有把数据一次性全部读写完(如读写缓冲区太小)，那么下次调用 epoll.poll()时，它还会通知你在上没读写完的文件描述符上继续读写，当然如果你一直不去读写，它会一直通知你！！！如果系统中有大量你不需要读写的就绪文件描述符，而它们每次都会返回，这样会大大降低处理程序检索自己关心的就绪文件描述符的效率！！！ 优点很明显：稳定可靠

Edge_triggered(边缘触发，有时也称状态触发)：当被监控的文件描述符上有可读写事件发生时，epoll.poll()会通知处理程序去读写。如果这次没有把数据全部读写完(如读写缓冲区太小)，那么下次调用epoll.poll()时，它不会通知你，也就是它只会通知你一次，直到该文件描述符上出现第二次可读写事件才会通知你！！！这种模式比水平触发效率高，系统不会充斥大量你不关心的就绪文件描述符！！！缺点：某些条件下不可靠

epoll实例：

import socket
import select
 
s = socket.socket()
s.bind(('127.0.0.1',8888))
s.listen(5)
epoll_obj = select.epoll()
epoll_obj.register(s,select.EPOLLIN)
connections = {}
while True:
 events = epoll_obj.poll()
 for fd, event in events:
  print(fd,event)
  if fd == s.fileno():
   conn, addr = s.accept()
   connections[conn.fileno()] = conn
   epoll_obj.register(conn,select.EPOLLIN)
   msg = conn.recv(200)
   conn.sendall('ok'.encode())
  else:
   try:
    fd_obj = connections[fd]
    msg = fd_obj.recv(200)
    fd_obj.sendall('ok'.encode())
   except BrokenPipeError:
    epoll_obj.unregister(fd)
    connections[fd].close()
    del connections[fd]
 
s.close()
epoll_obj.close()

import socket
 
flag = 1
s = socket.socket()
s.connect(('127.0.0.1',8888))
while flag:
 input_msg = input('input>>>')
 if input_msg == '0':
  break
 s.sendall(input_msg.encode())
 msg = s.recv(1024)
 print(msg.decode())
 
s.close()

The above is the detailed content of In-depth analysis of the select module in python. For more information, please follow other related articles on the PHP Chinese website!