Summary of usage of urllib2 module in Python network programming

1. The most basic application

import urllib2

url = r'http://www.baidu.com'
html = urllib2.urlopen(url).read()

print html

The client and the server communicate through request and response. The client first sends a request to the server, and then receives the response returned by the server

urllib2 provides the request class, which allows users to construct a request object before sending a request, and then send the request through the urllib2.urlopen method

import urllib2

url = r'http://www.baidu.com'
req = urllib2.Request(url)
html = urllib2.urlopen(req).read()
print html

Used first in the above example

req = urllib2.Request(url)

Instantiate a request object, then use

urllib2.urlopen(req)

Let’s open this page.

We noticed that when instantiating the Request object, the url is required, and there are several default parameters

Data and header are also commonly used in the base. Some websites that require login to browse often require these two parameters

import urllib 
import urllib2 
 
url = 'http://www.baidu.com/' 
values = {'name' : 'Michael Foord', 'location' : 'Northampton','language' : 'Python' } 
data = urllib.urlencode(values) 
req = urllib2.Request(url,data) 
response = urllib2.urlopen(req) 
the_page = response.read()

print the_page

This example is to send some data to Baidu. This example will return an error page, which is normal, because we do not need to post any information when visiting Baidu. However, an error will occur if we post it

Baidu will report an error if it cannot find the corresponding web page.

Of course this is POST data, it can also be used in the GET method, slightly modify the above code

Baidu queries through http://www.baidu.com/s?wd=XXX, so we need to urlencode the dictionary {'wd':'xxx'}

#coding:utf-8
import urllib 
import urllib2 
 
url = 'http://www.baidu.com/s' 
values = {'wd':'杨彦星'} 
data = urllib.urlencode(values)
print data 
url2 = url+'?'+data
response = urllib2.urlopen(url2) 
the_page = response.read()

print the_page

The following is an example of simulating login to Renren and then displaying the homepage content to explain the use of cookies in detail. The following is an example given in the document. We will modify this example to achieve the functions we want

import cookielib, urllib2
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
r = opener.open("http://example.com/")
 
#coding:utf-8
import urllib2,urllib
import cookielib

url = r'http://www.renren.com/ajaxLogin'

#创建一个cj的cookie的容器
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
#将要POST出去的数据进行编码
data = urllib.urlencode({"email":email,"password":pass})
r = opener.open(url,data)
print cj

When you see cj, it means you have visited the login page. You can’t tell whether you have logged in normally. You can check it by visiting http://www.renren.com/home

There are two points to explain in the above code. It took me a long time to understand it

r = opener.open(url,data) 

This sentence, why should we use the opener object to open instead of using utllib2 and urlopen? Not only is it written like this in the example, we write it this way. Through transformation, we can also use urllib2.urlopen. In fact, it is because opener is urllib2.bulid_opener Created, but you can understand it this way. After it was built, you did not install and use it, nor did it have its attributes and methods. If you want urllib2 to also have the attributes and methods of opener, you can first use urllib2.install_opener( opener) to "install" this opener. After installation, you can use urllib2 to operate it

#coding:utf-8
import urllib2,urllib
import cookielib

url = r'http://www.renren.com/ajaxLogin'

#创建一个cj的cookie的容器
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)
#将要POST出去的数据进行编码
data = urllib.urlencode({"email":email,"password":pass})
#r = opener.open(url,data)如果没有上面的urllib2.install_opener方法，就必须这样写了
r = urllib2.urlopen(url,data)
html = urllib2.urlopen('http://www.renren.com/home').read()

print html

Similarly, urllib2 also has proxy-related handles. The basic idea is similar to this.

2. Exception handling

URLError exception will be raised when urlopen() cannot process the response. The HTTPError exception is a subclass of URLError and is only raised when accessing an HTTP type URL.

1. URLError exception

The usual reasons for URLError are: no network connection (no route to the target server), and the accessed target server does not exist. In this case, the exception object will have a reason attribute (which is a tuple of (error code, error reason)).

#! /usr/bin/env python
#coding=utf-8
import urllib2

url="http://www.baidu.com/"
try:
 response=urllib2.urlopen(url)
except urllib2.URLError,e:
 print e.reason

2. HTTPError
Every HTTP response returned from the server has a status code. Among them, some status codes indicate that the server cannot complete the corresponding request. The default handler can handle some such status codes for us (if the returned response is a redirection, urllib2 will automatically obtain information from the redirected page for us) . There are some status codes that the urllib2 module cannot handle for us, so the urlopen function will cause an HTTPError exception, of which 404/401 is typical.
Instances of HTTPError exceptions have a code attribute of integer type, which represents the error status code returned by the server.
The default handler of the urllib2 module can handle redirects (status code is in the 300 range), and status codes in the 100-299 range indicate success. Therefore, the status code range that can cause HTTPError exceptions is: 400-599.
When an error occurs, the server returns an HTTP error code and error page. You can use the HTTPError instance as the return page, which means that the HTTPError instance not only has the code attribute, but also has methods such as read, geturl, and info.

#! /usr/bin/env python
#coding=utf-8
import urllib2

url="http://cs.scu.edu.cn/~duanlei"
try:
 response=urllib2.urlopen(url)
except urllib2.HTTPError,e:
 print e.code
 print e.read()

3. Summary
If you want to handle URLError and HTTPError in code, there are two methods, the code is as follows:

#! /usr/bin/env python
#coding=utf-8
import urllib2
url="xxxxxx" #需要访问的URL
try:
 response=urllib2.urlopen(url)
except urllib2.HTTPError,e: #HTTPError必须排在URLError的前面
 print "The server couldn't fulfill the request"
 print "Error code:",e.code
 print "Return content:",e.read()
except urllib2.URLError,e:
 print "Failed to reach the server"
 print "The reason:",e.reason
else:
 #something you should do
 pass #其他异常的处理
#! /usr/bin/env python
#coding=utf-8
import urllib2
url="http://xxx" #需要访问的URL
try:
 response=urllib2.urlopen(url)
except urllib2.URLError,e:
 if hasattr(e,"reason"):
 print "Failed to reach the server"
 print "The reason:",e.reason
 elif hasattr(e,"code"):
 print "The server couldn't fulfill the request"
 print "Error code:",e.code
 print "Return content:",e.read()
else:
 pass #其他异常的处理

In comparison, the second exception handling method is better.