How to use the urllib module for URL operations in Python 3.x

How to use the urllib module for URL operations in Python 3.x

Introduction

When doing network development, we often need to process URLs. Python provides the urllib module, in which the urllib.request submodule can conveniently perform URL operations. This article will introduce how to use urllib to open, read, download, upload and other operations of URLs.

Basic functions of the urllib.request module

The urllib.request module provides functions for interacting with URLs. Commonly used functions are:

• urlopen(url, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, *, cafile=None, capath=None, cadefault=False, context=None): Open a URL and return a file-like object. Optional parameters can specify POST data, timeout, certificate and other information.
• urlretrieve(url, filename=None, reporthook=None, data=None): Download the contents of the URL to the specified file.
• urlopen(url, data=None) and urlretrieve(url) are both synchronous and will block the process until the operation is completed.

Use urllib.request to open a URL

The following is a simple example showing how to use urlopen to open a URL and read the content of the web page.

import urllib.request

url = "https://www.example.com"
response = urllib.request.urlopen(url)
html = response.read().decode('utf-8')

print(html)

In this example, we first import the urllib.request module. Then, we define a URL, which is the address of a web page that needs to be opened. Next, use the urlopen function to open the URL and assign the returned file-like object to the response variable.

Use the response.read() function to read the content returned by the URL. To properly handle character encodings, we use decode('utf-8') to decode the content into a string. Finally, we print the resulting string.

Download File

Next, we will demonstrate how to download a file using the urlretrieve function.

import urllib.request

url = "https://www.example.com/example.png"
filename = "example.png"

urllib.request.urlretrieve(url, filename)
print("文件下载完成")

In this example, we also first import the urllib.request module. Then, we define a file URL that needs to be downloaded and save it in the variable url. Next, we define the file name to be saved after downloading and save it in the variable filename.

Finally, we use the urlretrieve function to download and save the file in the URL as the specified file name. After the download is completed, we print out a prompt message.

Use POST request

In addition to using GET request to open the URL, sometimes we also need to use POST request. Below is a sample code using a POST request.

import urllib.request
import urllib.parse

url = "https://www.example.com/example"
data = {"name": "Example", "age": 20}
data = urllib.parse.urlencode(data).encode('utf-8')

response = urllib.request.urlopen(url, data=data)
html = response.read().decode('utf-8')

print(html)

In this example, we also imported the urllib.request and urllib.parse modules. We have defined a URL that requires a POST request. Next, we define the data for the POST request, encode it into URL format using the urlendcode function and convert it into a byte stream.

Finally, we use the urlopen function and specify the requested URL and data. Likewise, we read the returned content and print it.

Using a proxy

In network access, sometimes we need to use a proxy. Below is a sample code using a proxy.

import urllib.request

url = "https://www.example.com"
proxy = {"http": "http://localhost:8080"}

proxy_handler = urllib.request.ProxyHandler(proxy)
opener = urllib.request.build_opener(proxy_handler)
urllib.request.install_opener(opener)

response = urllib.request.urlopen(url)
html = response.read().decode('utf-8')

print(html)

In this example, we define a URL and the address of the proxy. Next, we use the urllib.request.ProxyHandler function to create a proxy handler and pass in the proxy. We then create a custom opener using the urllib.request.build_opener function and set the proxy processor as its argument.

Finally, we use the urllib.request.install_opener function to install the custom opener as a global opener. In this way, each urlopen will use our custom opener to open the URL.

Summary

This article introduces how to use the urllib module for URL operations in Python 3.x. We used sample code to demonstrate how to open and read URL content, as well as how to download files, use POST requests, use proxies, and other operations. In actual development, you can use the functions provided by the urllib module to perform more complex URL operations according to specific needs.

The above is the detailed content of How to use the urllib module for URL operations in Python 3.x. For more information, please follow other related articles on the PHP Chinese website!