python - pyspider 集群报「connect to scheduler error 」
搭建集群 (一主一从，稳定运行后增加四台从，配置一致)后，正常运行
稳定运行6小时左右后，出现异常
点击 Active Tasks 后出现 connect to scheduler error 错误
重启 pyspider，未解决
回溯负载情况，均未见异常
查日志后发现大量在报
[W 160203 16:13:09 index:105] connect to scheduler rpc error: error(111, 'Connection refused')
向前查日志发现几次报错
[E 160203 16:08:30 scheduler:463]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/scheduler.py", line 458, in run
self.run_once()
File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/scheduler.py", line 444, in run_once
self._check_request()
File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/scheduler.py", line 267, in _check_request
task = self.on_new_request(task)
File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/scheduler.py", line 573, in on_new_request
self.put_task(task)
File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/scheduler.py", line 188, in put_task
exetime=_schedule.get('exetime', self.default_schedule['exetime'])
File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/task_queue.py", line 192, in put
self.priority_queue.put(task)
File "/usr/lib/python2.7/Queue.py", line 136, in put
self._put(item)
File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/task_queue.py", line 79, in _put
self.queue_dict[item.taskid] = item
MemoryError
数据库用的 MySQL，数据
1h: 59413
1d: 3096065
all: 2952765
目前发现可能和 taskdb 的量级有关，当前在 300万+ 的情况下出现了这个问题。但是从数据库相关监控信息看，CPU、连接数、网络等指标均未异常..
