How to Overcome Website Blocks Using Python's Requests and User Agents?

Faking Browser Visits with Python's Requests and User Agents: A Solution to Website Blocks

When accessing websites using Python's Requests package, you may encounter situations where the obtained HTML content differs significantly from that displayed in a browser. This is often due to the website employing blocks that identify and restrict access for non-browsers.

To overcome this, you can simulate browser visits by providing a User-Agent header, which identifies the type of browser and operating system being used. This allows the website to believe that it is a bona fide browser visit, granting access to the desired content. Here's how you can do it with Requests:

import requests

url = 'http://www.ichangtou.com/#company:data_000008.html'
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

response = requests.get(url, headers=headers)
print(response.content)

Alternatively, the fake-useragent package provides a convenient way to generate and use user agents for different browsers:

from fake_useragent import UserAgent
ua = UserAgent()
random_ua = ua.random
headers = {'User-Agent': random_ua}
response = requests.get(url, headers=headers)

By utilizing these techniques to fake browser visits, you can successfully access websites that previously blocked your Python-based attempts.

The above is the detailed content of How to Overcome Website Blocks Using Python's Requests and User Agents?. For more information, please follow other related articles on the PHP Chinese website!