Programming using tornado’s coroutine

After the release of tornado3, the concept of coroutine was strengthened. In asynchronous programming, it replaced the original gen.engine and became the current gen.coroutine. This decorator was originally designed to simplify asynchronous programming in tornado. Avoid writing callback functions, making development more consistent with normal logical thinking. A simple example is as follows:

class MaindHandler(web.RequestHandler):

@asynchronous

@gen.coroutine

def post(self):

client = AsyncHTTPClient()

resp = yield client .fetch(https://api.github.com/users")

if resp.code == 200:

resp = escape.json_decode(resp.body)

self.write(json.dumps(resp, indent=4, separators=(',', ':')))

        else:

            resp = {"message": "error when fetch something"}

                            ’ s ’ s to ’s ’ s ‐ ‐ ‐ ‐ ‐ self.write(json.dumps(resp, indent) =4, separators={',', ':')))

                                                                                                                                                                                                                 self.finish()

After the yield statement, ioloop will register the event and continue execution after resp returns. json.dumps is used here instead of escape.json_encode that comes with tornado, because when building a REST-style API, JSON format data is often accessed from the browser after formatting the data. , it will be more friendly when displayed on the browser. Github API is a user of this style. In fact, escape.json_encode is a simple packaging of json.dumps. When I submitted a pull request to package more functions, the author The answer is that escape does not intend to provide all json functions. Users can use the json module directly.

Gen.coroutine principle

In a previous blog, I talked about using the asynchronous features of tornado. An asynchronous library must be used. Otherwise, a single process will be blocked and the asynchronous effect will not be achieved at all. The most commonly used asynchronous library in Tornado is its own AsyncHTTPClient, as well as the OpenID login verification interface implemented on its basis. The library can be found here. Including the commonly used MongoDB Driver.

After version 3.0, the gen.coroutine module becomes more prominent. The coroutine decorator can make asynchronous programming that relies on callbacks look like synchronous programming. Among them is the Send function of the generator in Python. In generators, the yield keyword is often compared to return in a normal function. It can be used as an iterator, so that next() can be used to return the result of yield. But there is another way to use the generator, which is to use the send method. Inside the generator, the result of yield can be assigned to a variable, and this value is sent through the external generator client. Give an example:

def test_yield():

pirnt "test yeild"

says = (yield)

print says

if __name__ == "__main__":

client = test_yield ()

client.next()

client.send("hello world")

The output results are as follows:

test yeild

hello world

The function that is already running will hang until called Its client uses the send method, and the original function continues to run. The gen.coroutine method here is to execute the required operations asynchronously, and then wait for the result to be returned before sending it to the original function, which will continue to execute. In this way, the code written in a synchronous manner achieves the effect of asynchronous execution.

Tornado asynchronous programming

Use coroutine to implement asynchronous programming with function separation. The details are as follows:

@gen.coroutine

def post(self):

client = AsyncHTTPClient()

resp = yield client.fetch("https://api.github.com/users")

if resp == 200:

body = escape.json_decode(resy.body)

else:

body = {"message": "client fetch error"}

logger.error("client fetch error% d, %s" % (resp.code, resp.message))

self.write(escape.json_encode(body))

self.finish()

After changing to a function, it can become like this;

@gen.coroutime

def post(self):

resp = yield GetUser()

self.write(resp)

@gen.coroutine

def GetUser():

client = AsyncHTTPClient()

resp = yield client.fetch("https://api.github.com/users")

if resp.code == 200:

resp = escape.json_decode(resp.body)

Else: s resp = {"message": "fetch client error"}

logger.error ("client fetch error %d, %s" %(resp.message)) (resp)

Here, when asynchronous is encapsulated in a function, instead of using the return keyword to return like a normal program, the gen module provides a gen.Return method. This is achieved through the raise method. This is also related to the fact that it is implemented using a generator.

Use coroutine to run scheduled tasks

There is such a method in Tornado:

tornado.ioloop.IOLoop.instance().add_timeout()

This method is a non-extension of time.sleep Blocking version, which accepts two parameters: a length of time and a function. Indicates the amount of time after which the function will be called. Here it is based on ioloop and therefore non-blocking. This method is often used in client long connection and callback function programming. But using it to run some scheduled tasks is helpless. Usually there is no need to use it when running scheduled tasks. But when I was using heroku, I found that if I didn't register a credit card, I could only use the hosting of a simple Web Application. Cannot add scheduled tasks to run. So I came up with such a method. Here, I mainly use it to grab data through the Github API interface at intervals. The usage method is as follows:

decorator

def sync_loop_call(delta=60 * ​​1000):

"""

Wait for func down then process add_timeout

"""

def wrap_loop(func):

                                                                                                                                                                                                                    r start at %d" %

_ (Func .__ name__, int (time.time ()))

Try:

yield func (*args, ** kwargs)

except exception, e:

options.logger.error ("function % % r error: %s" %

                                                                                                                                                                              ​                                  (func.__name__, int(time.time()) :)

                                                                   return wrap_func

                                                            Return wrap_loop

Task function

 @sync_loop_call (delta=10 * 1000)

def worker():

"""

Do something

"""

Add task

if __name__ == "__main__":

worker()

app.listen(options.port)

tornado.ioloop.IOLoop.instance().start()

After doing this, when the Web Application starts, the scheduled task will start running, and because it is It is event-based and executed asynchronously, so it does not affect the normal operation of the Web service. Of course, the task cannot be blocking or computationally intensive. I mainly capture data here, and I use the asynchronous capture method that comes with Tornado.

In the sync_loop_call decorator, I added the @gen.coroutine decorator to the wrap_func function, which ensures that only after the yeild function is executed, the add_timeout operation will be executed. Without @gen.coroutine decorator. Then add_timeout will be executed without waiting for yeild to return.

For complete examples, please see my Github. This project is built on heroku. Used to display Github user activity rankings and user regional distribution. You can visit Github-Data to view. Since heroku is blocked in China, you need to climb over the wall to access it.

Summary

Tornado is a non-blocking web server and web framework, but when used, only asynchronous libraries can truly take advantage of its asynchronous advantages. Of course, sometimes because the App itself requires It's not very high, and if the blockage isn't particularly severe, it won't be a problem. In addition, when using the coroutine module for asynchronous programming, when a function is encapsulated into a function, even if an error occurs while the function is running, it will not be thrown if it is not caught, which makes debugging very difficult.