How to Set Custom User Agents in Python urllib2 for Web Scraping?

Setting User Agents in urllib2

While using urllib2.urlopen for web scraping, the default user agent is typically the Python package name. This may not always be desirable, as websites can enforce restrictions based on the user agent. To modify the user agent, one can utilize a custom urllib2 opener.

In Python 2.x, urllib2.build_opener can be employed to create a new opener with custom headers, including the user agent. The following code snippet demonstrates how to change the user agent to Mozilla:

opener = urllib2.build_opener()
opener.addheaders = [('User-Agent', 'Mozilla/5.0')]
response = opener.open('http://www.stackoverflow.com')

In Python 3, the equivalent to urllib2.build_opener is urllib.request.build_opener. The same code structure can be used to set custom user agents in Python 3 as well.

The above is the detailed content of How to Set Custom User Agents in Python urllib2 for Web Scraping?. For more information, please follow other related articles on the PHP Chinese website!