How to build an anonymous proxy pool using Python? How to build an anonymous proxy pool

The content of this article is to introduce how to use Python to build an anonymous proxy pool? How to build an anonymous proxy pool. It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.

I often hear many people complain that their IP is blocked by the website due to too many crawlers, and they have to frequently use various proxy IPs, but because most of the public proxies on the Internet cannot be used, and You have to spend money and energy to apply for a VIP agent, only to be blocked after several twists and turns. Hereby write an article on how to use Python to build a proxy pool to reduce time and energy costs and realize the function of automatically obtaining active proxy IPs.

Principle of operation

1. Website proxy acquisition

1. Climb the IP list of the free proxy website to test whether it is available and whether it is high Anonymous

2. If they are all, put them into the database, otherwise discard them.

3. Repeat step 2

2. Ensure that failed agents can be picked out from the agent pool as soon as possible

1. Get the IP from the crawler database

2. Test the availability and anonymity of the IP

3. If available and anonymous, keep it , otherwise discard.

4. Repeat step 1

Instructions ①: You can create a crawler daemon (Daemon), which has this aspect Friends who need it can google it by themselves, I won’t introduce it here.

Explanation ②: You can establish an external proxy information interface. It doesn’t matter whether you use NodeJS or Flask/Django or PHP to write it. I won’t introduce it here.

Implementation:

Recommended libraries: requests, BeautifulSoup, re, sqlite3.

Among them, the requests library is used to obtain the agent website page, the BeautifulSoup and re libraries are used to obtain agent information, and sqlite3 is used to access this information.

If necessary (such as when the proxy website has an anti-crawler strategy), PhantomJS can be used to replace requests, or the corresponding library can be used for data cleaning (such as base64 decoding).

The following briefly shows the code of each part:

The first is to select multiple websites that can crawl proxies and are not easily blocked by IP. Here, proxy-list.org is used as an example:

BASE_URL = "https://proxy-list.org/english/index.php?p="


#IP地址及端口的正则

Re_Pattern_IP = re.compile("(.*):")

Re_Pattern_PORT = re.compile(":(.*)")


#网站有11页，所以循环11次获取所有代理IP及端口

for startingURL_Param in range(1,11):

   HTML_ProxyPage = requests.get(BASE_URL+str(startingURL_Param)).content

   soup = bs(HTML_ProxyPage,"html.parser")

   for Raw_ProxyInfo in soup.find_all("ul",{"class":None}):

       #此网站有用Base64简单对代理进行了加密，所以这里对其解码

       ip_port = base64.b64decode(Raw_ProxyInfo.find("li",{"class":"proxy"}).text.replace("Proxy('","").replace("')",""))

       #接下来利用正则从网页数据中提取我们需要的信息

       IP = re.findall(Re_Pattern_IP, ip_port)[0]

       PORT = re.findall(Re_Pattern_PORT, ip_port)[0]

       TYPE = Raw_ProxyInfo.find("li",{"class":"https"}).text

The following is a code for a simple proxy pool framework class, which provides the addition, deletion, connectability detection, and anonymity detection of the proxy database:

class ProxyPool:        

   #初始化爬虫池数据库

   def __init__(self,ProxyPoolDB):

       self.ProxyPoolDB = ProxyPoolDB

       self.conn = sqlite3.connect(self.ProxyPoolDB, isolation_level=None)

       self.cursor = self.conn.cursor()

       self.TB_ProxyPool = "TB_ProxyPool"

       self.cursor.execute("CREATE TABLE IF NOT EXISTS "+self.TB_ProxyPool+"(ip TEXT UNIQUE, port INTEGER, protocol TEXT)")

 

   #添加代理IP进代理池的接口

   def addProxy(self, IP, PORT, PROTOCOL):  

       self.cursor.execute("INSERT OR IGNORE INTO " + self.TB_ProxyPool+"(ip, port, protocol) VALUES (?,?,?)", [IP,PORT,PROTOCOL])

 

   #检查代理的匿名性及可连接性

   def testConnection(self, IP, PORT, PROTOCOL):

       proxies = { PROTOCOL: IP+":"+PORT }

       try:

           OrigionalIP = requests.get("http://icanhazip.com",timeout=REQ_TIMEOUT).content

           MaskedIP = requests.get("http://icanhazip.com", timeout=REQ_TIMEOUT,proxies=proxies).content

           if OrigionalIP != MaskedIP:

               return True

           else:

               return False

       except:    

           return False

 

   #删除代理IP对应的数据库记录

   def delRecord(self, IP):

       self.cursor.execute("DELETE FROM "+self.TB_ProxyPool+" WHERE ip=?",(IP,))

The following is to delete the proxy pool. Code for invalid IP: Looking back, the logic is not rigorous enough, various functions are too coupled, and many paragraphs need to be rewritten. Because the code is run on the campus network, the stability of the network connection needs to be taken into consideration, which causes inconsistencies between some codes. Chaotic relationships. The method of detecting proxy anonymity through icanhazip.com may be effective, but it ignores the X-Forwarded-For HTTP header, so it is very risky and must be improved.

Verifying the validity of the agents in the agent pool requires multi-threading, and the current solution is too inefficient.
Complete code

What is placed in this article is the core code of the proxy pool, aiming to provide readers with ideas and references that can be implemented by themselves, Ubuntu 16.04 and Kali The test can be run with Python 2.7.

The above is the detailed content of How to build an anonymous proxy pool using Python? How to build an anonymous proxy pool. For more information, please follow other related articles on the PHP Chinese website!