Python asynchronously calls command line tools

　When you adopt an asynchronous programming method based on event loops in your own Python program, you will find yourself unconsciously attracted to it, not because this method is great, but because you have to think about it. The method ensures that no link in the program is blocked!

For example, the current scenario is to read every piece of unprocessed data from MongoDB, download and save the image information, and then update the contents of the database. The commonly used MongoDB asynchronous driver in Python is Motor:

Used in combination with asyncio as follows: 　

import motor.motor_asyncio
　　import asyncio
　　client = motor.motor_asyncio.AsyncIOMotorClient()
　　db = client.test_database
　　async def run():
　　async for mm in db.test_database.find({"status": 0}):
　　print(mm['img_src'])
　　# Download Image Here
　　# dl_img(mm['img_src'])
　　await db.test_database.update({"_id": mm['_id']}, {"$set": {"status":1}})
　　loop = asyncio.get_event_loop()
　　loop.run_until_complete(run())

At this time, if the operation at dl_img() is blocked, then asynchronous processing is meaningless. Of course, you can still use the asynchronous network request library aiohttp to achieve image downloading:

　　async with session.get(img) as resp:
　　with open(img.split("/")[-1], 'wb') as fd:
　　while True:
　　chunk = await resp.content.read(1024)
　　if not chunk:
　　break
　　fd.write(chunk)

Of course, you can also directly call the system command line tool (such as wget) to complete the download task without downloading it yourself. Python implements system command calls through the subprocess standard library (replacing the old os.system (cmd)). To perform a download task, you only need:

import subprocess as sb
sb.run(['wget', img], shell=True)

However, this calling method cannot be used directly in the asyncio event loop, but asyncio provides the corresponding subprocess interface:

asyncio.create_subprocess_exec(*args, ...)
asyncio.create_subprocess_shell(cmd, ...)

Both methods return an asyncio.subprocess.Process instance, and its interface design completely imitates subprocess.Popen (the underlying implementation of subprocess.run() mentioned above ), so it is easy to transplant its usage into the event loop: 　

async def dl_img(src):
　　dl = await asyncio.create_subprocess_shell('wget {} -O {}'.format(src, src.split("/")[-1])
　　await dl.wait()

In addition to the usage in the above scenario, you can also directly put the execution of the command line as a task into the event loop:

　　loop = asyncio.get_event_loop()
　　sb = asyncio.create_subprocess_shell('exit 7', loop=loop)
　　proc = loop.run_until_complete(sb)
　　exitcode = loop.run_until_complete(proc.wait())

Summary

The meaning of asynchronous programming in Python is not to let the CPU block IO, so you need to pay attention to using the correct asynchronous method in every blocking operation. Once these operations are encapsulated into asynchronous Tasks, There is no need to worry about subsequent scheduling execution.