python - tornado怎么处理非常耗时的定时任务？

Question

代码大致如下：

class MainHandler(RequestHandler):
    def get(self):
        ...
        ...
        
        
def update():
    ...
    ...
    IOLoop.instance().add_timeout(time.time() + 3600, update)
    
   
if __name__ == '__main__':
    application = tornado.web.Application([
        ....
    ])
    server = HTTPServer(application)
    server.start(2) # 开启两个进程
    IOLoop.instance().add_timeout(time.time() + 3600, update)  # 耗时的定时update
    ...
    ...

其中update()是一个特别耗时的任务
目前的做法是加大add_timeout的时间间隔，以避免两个进程全都去做update了

因为对异步编程并不熟，所以想请教一下有没有更合适的做法

Answer 1

It sounds like this task will occupy the main thread for a long time and the specific time is not sure, so using coroutines is obviously not appropriate. It is recommended to try ThreadPoolExecutor, and then use yield threadPool.submit(fn)

Answer 2

tornado comes with its own gen module, and its official website documentation has clear examples

It can be coroutineized. (With callbacks, the logic is complicated and it is tiring to read the code; for threads/processes, scheduling issues need to be considered)

Answer 3

It’s useless. . If you don't consider anything else, if you have a single thread and spends too long in one place, it is really blocked.
Refer to the code below:

class TestHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    @tornado.gen.coroutine
    def get(self):
        yield gen.Task(self.sleep)
        self.write('ok')
        self.finish()

    def sleep(self, callback):
        for i in range(100000000):
            if i % 100000 == 0:
                pass
        print('ooo')
        callback()

Answer 4

It depends on whether your task is network IO intensive or local IO intensive/CPU intensive. If it is the former, just use PeriodicCallback. If it is the latter, it is recommended to implement a queue and then write a worker to run the task alone