如题描述
因为一些需求想对一份url列表进行get请求扫描,看看这些页面是否存在
平时用的requests模块的get和多进程来实现,感觉还是不够快.于是产生了如题描述的这种想法.
顺便问下,为什么vps提供商会封我...好郁闷.
高洛峰2017-04-18 10:23:17
I usually use the get and multi-process implementation of the requests module, but it still feels not fast enough. So I came up with the idea described in the title.
Depending on a variety of reasons, it won't necessarily be faster if you discard the GET response body.
If HEAD meets your requirements, use HEAD.
Remember to use requests.Session
, so that access to the same server can be more than doubled.
PS: It is recommended to learn HTTP carefully, so that when you encounter problems, you don’t need to ask and wait, and finally get a solution that you don’t understand.