怪我咯2017-04-18 10:14:48
Changing it to asynchronous is not of much use. Your situation is very computationally intensive, and your time should be spent on "key in need_cols". need_cols should also be a large array. Just find a way to optimize this area.
Try changing it to this
ind = set(i.keys()) & set(need_cols)
temp = {d: i[d] for d in ind}
高洛峰2017-04-18 10:14:48
Ask and answer your own questions
Just change it to a generator
def data_generator(*args):
for i in v_data: # v_data是个较大的dict
temp = {key: value for key, value in i.iteritems() if key in need_cols} # 部分key不需要
temp["sid"] = sid # 单独加一个sid
yield temp