python - pyspider 集群报「connect to scheduler error 」

搭建集群 (一主一从，稳定运行后增加四台从，配置一致)后，正常运行
稳定运行6小时左右后，出现异常
点击 Active Tasks 后出现 connect to scheduler error 错误
重启 pyspider，未解决
回溯负载情况，均未见异常

查日志后发现大量在报

[W 160203 16:13:09 index:105] connect to scheduler rpc error: error(111, 'Connection refused')

向前查日志发现几次报错

[E 160203 16:08:30 scheduler:463]

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/scheduler.py", line 458, in run
    self.run_once()
  File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/scheduler.py", line 444, in run_once
    self._check_request()
  File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/scheduler.py", line 267, in _check_request
    task = self.on_new_request(task)
  File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/scheduler.py", line 573, in on_new_request
    self.put_task(task)
  File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/scheduler.py", line 188, in put_task
    exetime=_schedule.get('exetime', self.default_schedule['exetime'])
  File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/task_queue.py", line 192, in put
    self.priority_queue.put(task)
  File "/usr/lib/python2.7/Queue.py", line 136, in put
    self._put(item)
  File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/task_queue.py", line 79, in _put
    self.queue_dict[item.taskid] = item
MemoryError

数据库用的 MySQL，数据
1h: 59413
1d: 3096065
all: 2952765

目前发现可能和 taskdb 的量级有关，当前在 300万+ 的情况下出现了这个问题。但是从数据库相关监控信息看，CPU、连接数、网络等指标均未异常..

ringa_lee2813日前962

python - pyspider 集群报「connect to scheduler error 」

全員に返信(1)返信します