python - scrapy的异常处理一般怎么做?

Question

发现在运行过程中有少量的请求有异常，或者请求响应了404 500之类的，又或者代码本身运行抛异常， 怎么记录这些异常呢。

Answer 1

我目前的做法是用下载中间件去捕捉失败的请求，仅供大家参考

class CustomFaillogMiddleware(object):

    @classmethod
    def from_crawler(cls, crawler):
        return cls()

    def process_response(self, request, response, spider):
        if response.status >= 400:
            reason = response_status_message(response.status)
            self._faillog(request, u'HTTPERROR',reason, spider)
        return response

    def process_exception(self, request, exception, spider):
        self._faillog(request, u'EXCEPTION', exception, spider)
        return request

    def _faillog(self, request, errorType, reason, spider):
        with codecs.open('log/faillog.log', 'a', encoding='utf-8') as file:
            file.write("%(now)s [%(error)s] %(url)s reason: %(reason)s \r\n" %
                       {'now':datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
                        'error': errorType,
                        'url': request.url,
                        'reason': reason})

Answer 2

你是想通过log的方式来记录异常吗？ Scrapy提供了log功能的

Answer 3

http://scrapy-chs.readthedocs.io/zh_CN/1.0/topics/logging.html