Python Tutorial

How to prevent IP being blocked by python crawler

little bottle

Apr 10, 2019 pm 05:06 PM

pythonreptile

When writing a crawler to crawl data, especially when crawling a large amount of data, because many websites have anti-crawler measures, it is easy to have their IP blocked and cannot continue to crawl. This article summarizes some countermeasures on how to solve this problem. These measures can be used alone or at the same time for better results.

Fake User-Agent

Set the User-Agent in the request header to the User-Agent in the browser to fake browser access. For example:

headers = {&#39;User-Agent&#39;:&#39;Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36&#39;}
resp = requests.get(url,headers = headers)

Set a random time interval between each repeated crawling

# 比如：
time.sleep(random.randint(0,3))  # 暂停0~3秒的整数秒，时间区间：[0,3]
# 或：
time.sleep(random.random())  # 暂停0~1秒，时间区间：[0,1)

Fake cookies

If you can access a page normally from the browser, you can Copy the cookies in the browser and use them, for example:

cookies = dict(uuid=&#39;b18f0e70-8705-470d-bc4b-09a8da617e15&#39;,UM_distinctid=&#39;15d188be71d50-013c49b12ec14a-3f73035d-100200-15d188be71ffd&#39;)
resp = requests.get(url,cookies = cookies)

# 把浏览器的cookies字符串转成字典
def cookies2dict(cookies):
    items = cookies.split(&#39;;&#39;)
    d = {}
    for item in items:
        kv = item.split(&#39;=&#39;,1)
        k = kv[0]
        v = kv[1]
        d[k] = v
    return d

Note: After using browser cookies to initiate a request, if the request frequency is too frequent, the IP will still be blocked. At this time, you can perform the corresponding actions on the browser. Manual verification (such as clicking on the verification image, etc.), and then you can continue to use the cookie to initiate requests normally.

Use proxy

You can use multiple proxy IPs for access to prevent the same IP from launching too many requests and being blocked, such as:

proxies = {&#39;http&#39;:&#39;http://10.10.10.10:8765&#39;,&#39;https&#39;:&#39;https://10.10.10.10:8765&#39;}
resp = requests.get(url,proxies = proxies)
# 注：免费的代理IP可以在这个网站上获取：http://www.xicidaili.com/nn/

[Recommended courses ：Python video tutorial】

The above is the detailed content of How to prevent IP being blocked by python crawler. For more information, please follow other related articles on the PHP Chinese website!

Statement

This article is reproduced at:博客园. If there is any infringement, please contact admin@php.cn delete

What are the alternatives to concatenate two lists in Python?May 09, 2025 am 12:16 AM

There are many methods to connect two lists in Python: 1. Use operators, which are simple but inefficient in large lists; 2. Use extend method, which is efficient but will modify the original list; 3. Use the = operator, which is both efficient and readable; 4. Use itertools.chain function, which is memory efficient but requires additional import; 5. Use list parsing, which is elegant but may be too complex. The selection method should be based on the code context and requirements.

Python: Efficient Ways to Merge Two ListsMay 09, 2025 am 12:15 AM

There are many ways to merge Python lists: 1. Use operators, which are simple but not memory efficient for large lists; 2. Use extend method, which is efficient but will modify the original list; 3. Use itertools.chain, which is suitable for large data sets; 4. Use * operator, merge small to medium-sized lists in one line of code; 5. Use numpy.concatenate, which is suitable for large data sets and scenarios with high performance requirements; 6. Use append method, which is suitable for small lists but is inefficient. When selecting a method, you need to consider the list size and application scenarios.

Compiled vs Interpreted Languages: pros and consMay 09, 2025 am 12:06 AM

Compiledlanguagesofferspeedandsecurity,whileinterpretedlanguagesprovideeaseofuseandportability.1)CompiledlanguageslikeC arefasterandsecurebuthavelongerdevelopmentcyclesandplatformdependency.2)InterpretedlanguageslikePythonareeasiertouseandmoreportab

Python: For and While Loops, the most complete guideMay 09, 2025 am 12:05 AM

In Python, a for loop is used to traverse iterable objects, and a while loop is used to perform operations repeatedly when the condition is satisfied. 1) For loop example: traverse the list and print the elements. 2) While loop example: guess the number game until you guess it right. Mastering cycle principles and optimization techniques can improve code efficiency and reliability.

Python concatenate lists into a stringMay 09, 2025 am 12:02 AM

To concatenate a list into a string, using the join() method in Python is the best choice. 1) Use the join() method to concatenate the list elements into a string, such as ''.join(my_list). 2) For a list containing numbers, convert map(str, numbers) into a string before concatenating. 3) You can use generator expressions for complex formatting, such as ','.join(f'({fruit})'forfruitinfruits). 4) When processing mixed data types, use map(str, mixed_list) to ensure that all elements can be converted into strings. 5) For large lists, use ''.join(large_li

Python's Hybrid Approach: Compilation and Interpretation CombinedMay 08, 2025 am 12:16 AM

Pythonusesahybridapproach,combiningcompilationtobytecodeandinterpretation.1)Codeiscompiledtoplatform-independentbytecode.2)BytecodeisinterpretedbythePythonVirtualMachine,enhancingefficiencyandportability.

Learn the Differences Between Python's 'for' and 'while' LoopsMay 08, 2025 am 12:11 AM

ThekeydifferencesbetweenPython's"for"and"while"loopsare:1)"For"loopsareidealforiteratingoversequencesorknowniterations,while2)"while"loopsarebetterforcontinuinguntilaconditionismetwithoutpredefinediterations.Un

Python concatenate lists with duplicatesMay 08, 2025 am 12:09 AM

In Python, you can connect lists and manage duplicate elements through a variety of methods: 1) Use operators or extend() to retain all duplicate elements; 2) Convert to sets and then return to lists to remove all duplicate elements, but the original order will be lost; 3) Use loops or list comprehensions to combine sets to remove duplicate elements and maintain the original order.

See all articles

Hot AI Tools

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress images for free

Clothoff.io

AI clothes remover

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

How to fix KB5055523 fails to install in Windows 11?

4 weeks agoByDDD

How to fix KB5055518 fails to install in Windows 10?

4 weeks agoByDDD

Roblox: Grow A Garden - Complete Mutation Guide

3 weeks agoByDDD

Roblox: Bubble Gum Simulator Infinity - How To Get And Use Royal Keys

3 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

How to fix KB5055612 fails to install in Windows 10?

3 weeks agoByDDD

Hot Tools

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software