首页 > 文章 > 后端开发 > Python爬虫抓取代理IP并检验可用性的实例

Python爬虫抓取代理IP并检验可用性的实例

不言原创: 2018-05-07 12:00:341799浏览

这篇文章主要介绍了关于Python爬虫抓取代理IP并检验可用性的实例，有着一定的参考价值，现在分享给大家，有需要的朋友可以参考一下

经常写爬虫，难免会遇到ip被目标网站屏蔽的情况，银次一个ip肯定不够用，作为节约的程序猿，能不花钱就不花钱，那就自己去找吧，这次就写了下抓取西刺代理上的ip，但是这个网站也反爬！！！

至于如何应对，我觉得可以通过增加延时试试，可能是我抓取的太频繁了，所以被封IP了。

但是，还是可以去IP巴士试试的，条条大路通罗马嘛，不能吊死在一棵树上。

不废话，上代码。

#!/usr/bin/env python
# -*- coding:utf8 -*-
import urllib2
import time
from bs4 import BeautifulSoup
import sys
reload(sys)
sys.setdefaultencoding( "utf-8" )
req_header = {&#39;User-Agent&#39;:&#39;Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11&#39;,
 &#39;Accept&#39;:&#39;text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8&#39;,
 #&#39;Accept-Language&#39;: &#39;en-US,en;q=0.8,zh-Hans-CN;q=0.5,zh-Hans;q=0.3&#39;,
 &#39;Accept-Charset&#39;:&#39;ISO-8859-1,utf-8;q=0.7,*;q=0.3&#39;,
 &#39;Accept-Encoding&#39;:&#39;en-us&#39;,
 &#39;Connection&#39;:&#39;keep-alive&#39;,
 &#39;Referer&#39;:&#39;http://www.baidu.com/&#39;
 }
req_timeout = 5
testUrl = "http://www.baidu.com/"
testStr = "wahaha"
file1 = open(&#39;proxy.txt&#39; , &#39;w&#39;)
# url = ""
# req = urllib2.Request(url,None,req_header)
# jsondatas = urllib2.urlopen(req,None,req_timeout).read()
cookies = urllib2.HTTPCookieProcessor()
checked_num = 0
grasp_num = 0
for page in range(1, 160):
 req = urllib2.Request(&#39;http://www.xici.net.co/nn/&#39; + str(page), None, req_header)
 html_doc = urllib2.urlopen(req, None, req_timeout).read()
 # html_doc = urllib2.urlopen(&#39;http://www.xici.net.co/nn/&#39; + str(page)).read()
 soup = BeautifulSoup(html_doc)
 trs = soup.find(&#39;table&#39;, id=&#39;ip_list&#39;).find_all(&#39;tr&#39;)
 for tr in trs[1:]:
  tds = tr.find_all(&#39;td&#39;)
  ip = tds[1].text.strip()
  port = tds[2].text.strip()
  protocol = tds[5].text.strip()
  if protocol == &#39;HTTP&#39; or protocol == &#39;HTTPS&#39;:
   #of.write(&#39;%s=%s:%s\n&#39; % (protocol, ip, port))
   print &#39;%s=%s:%s&#39; % (protocol, ip, port)
   grasp_num +=1
   proxyHandler = urllib2.ProxyHandler({"http": r&#39;http://%s:%s&#39; % (ip, port)})
   opener = urllib2.build_opener(cookies, proxyHandler)
   opener.addheaders = [(&#39;User-Agent&#39;,
         &#39;Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36&#39;)]
   t1 = time.time()
   try:
    req = opener.open(testUrl, timeout=req_timeout)
    result = req.read()
    timeused = time.time() - t1
    pos = result.find(testStr)
    if pos > 1:
     file1.write(protocol+"\t"+ip+"\t"+port+"\n")
     checked_num+=1
     print checked_num, grasp_num
    else:
     continue
   except Exception,e:
    continue
file1.close()
print checked_num,grasp_num

个人感觉代码里没有太复杂的，就没有加注释，相信大家基本可以理解，如有问题也请多批评指正，共同进步！

Python爬虫抓取代理IP并检验可用性的实例

相关文章