python - Scrapy 总是出现卡顿的情况

Question

最近发现部分服务器上出现几个情况：

请教scrapy大神，

1. scrapy在执行scrapy crawl price 后总是停顿十几秒才会打印出下一步的输出结果。

2. 自动设置use-agent，会卡顿十几秒换下一个use-agent继续卡住。

有没有大神读过源码，知道这是什么情况吗？

Answer 1

I won’t encounter long-term lags at the beginning, but long-term lags will occur after a long period of time.

Answer 2

There is nothing wrong with this and the source code. You have to take a look at your code, or post the spider code to take a look. Only then can we know the specific problem

Answer 3

At least when I’m crawling data now, there’s no lag like you mentioned. There will only be a slight lag at the beginning of startup. I guess it is because there are too many files loaded. Your lagging situation. You have to see if it's because of the crawler you wrote. Please post the code.