What is the use of coroutines and concurrency in python?

This chapter will introduce you to the role of coroutines and concurrency in python, so that you can understand the pros and cons of using coroutines, and the role of the gevent concurrency framework. It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.

Coroutine

Coroutine is a lightweight thread in user mode, also known as micro-thread.

The coroutine has its own register context and stack. When the schedule is switched, the register context and stack are saved elsewhere. When switching back, the previously saved register context and stack are restored. Therefore: the coroutine can retain the state of the last call (that is, a specific combination of all local states). Each time the process re-enters, it is equivalent to entering the state of the last call. In other words: entering the state when it left last time. The location of the logical flow.

Advantages:

1. No overhead of thread context switching

2. No overhead of atomic operation locking and synchronization

3. Conveniently switch control flow and simplify the programming model

4. High concurrency, high scalability and low cost: it is not a problem for a CPU to support tens of thousands of coroutines. So it is very suitable for high concurrency processing.

The so-called atomic operation refers to an operation that will not be interrupted by the thread scheduling mechanism; once this operation starts, it will run until the end without any context switch (switch to another thread).

Atomic operations can be one step or multiple steps, but the order cannot be disrupted or only the executed part can be cut off. Seeing as a whole is at the heart of atomicity.

Disadvantages:

1. Unable to utilize multi-core resources: The essence of the coroutine is a single thread. It cannot use multiple cores of a single CPU at the same time. The coroutine needs to be Processes must cooperate to run on multiple CPUs. Of course, most of the applications we write daily do not have this necessity, unless they are CPU-intensive applications.

2. Performing blocking operations (such as IO) will block the entire program

Use Gevent

gevent is a concurrency framework for python, with micro-thread greenlet as the core, using the epoll event listening mechanism and many other optimizations to become efficient.:

• Simple example

gevent's sleep can hand over control. When we use gevent in functions that are restricted by network or IO, these functions will be scheduled cooperatively. The true capabilities of gevent will be unleashed. Gevent handles all the details to ensure that your network library will implicitly hand over execution rights to the greenlet context when possible.

import gevent
 
def foo():
    print('running in foo')
    gevent.sleep(0)
    print('com back from bar in to foo')
 
def bar():
    print('running in bar')
    gevent.sleep(0)
    print('com back from foo in to bar')
 
# 创建线程并行执行程序
gevent.joinall([
    gevent.spawn(foo),
    gevent.spawn(bar),
])

Execution result:

• Synchronous asynchronous

import random
import gevent
 
def task(pid):
    gevent.sleep(random.randint(0, 2) * 0.001)
    print('Task %s done' % pid)
 
def synchronous():
    for i in range(1, 10):
        task(i)
 
def asynchronous():
    threads = [gevent.spawn(task, i) for i in range(10)]
    gevent.joinall(threads)
 
print('Synchronous:')
synchronous()
 
print('Asynchronous:')
asynchronous()

Execution output:

• Using coroutines in a subclass method

You can subclass the Greenlet class. Overload its _run method, similar to multi-thread and multi-process modules

import gevent
from gevent import Greenlet
 
class Test(Greenlet):
 
    def __init__(self, message, n):
        Greenlet.__init__(self)
        self.message = message
        self.n = n
 
    def _run(self):
        print(self.message, 'start')
        gevent.sleep(self.n)
        print(self.message, 'end')
 
tests = [
    Test("hello", 3),
    Test("world", 2),
]
 
for test in tests:
    test.start()  # 启动
 
for test in tests:
    test.join()  # 等待执行结束

• Use monkey patch to modify the system standard library (automatically switch coroutines)

When a greenlet encounters an IO operation, such as accessing the network, it will automatically switch to other greenlets, wait until the IO operation is completed, and then switch back at the appropriate time to continue execution.

Since IO operations are very time-consuming, the program is often placed in a waiting state. With gevent automatically switching coroutines for us, it is guaranteed that greenlets are always running instead of waiting for IO.

Since switching is automatically completed during IO operations, gevent needs to modify some of the standard libraries that come with Python. This process is completed through monkey patch at startup

import gevent
import requests
from gevent import monkey
 
monkey.patch_socket()
 
def task(url):
    r = requests.get(url)
    print('%s bytes received from %s' % (len(r.text), url))
 
gevent.joinall([
    gevent.spawn(task, 'https://www.baidu.com/'),
    gevent.spawn(task, 'https://www.qq.com/'),
    gevent.spawn(task, 'https://www.jd.com/'),
])

Execution output:

It can be seen that the three network operations are executed concurrently, and the ending order is different

The above is the detailed content of What is the use of coroutines and concurrency in python?. For more information, please follow other related articles on the PHP Chinese website!