Fetching Data from an HTTP API with Python

Python efficiently accesses HTTP API: requests library and request cache

This article is excerpted from "Practical Python", and the author Stuart demonstrates how to easily access the HTTP API using Python and several third-party modules.

Most cases, processing third-party data requires access to the HTTP API, i.e., sending HTTP requests to web pages designed to be read by machines rather than manually. API data is usually in a machine-readable format, usually in JSON or XML. Let's see how to access the HTTP API using Python.

The basic principles of using HTTP API are simple:

1. Send an HTTP request to the API's URL, which may include some authentication information (such as API keys) to prove that we are authorized.
2. Get data.
3. Use data to complete useful operations.

The Python standard library provides enough functions to do all of this without any additional modules, but it will make our work easier if we use several third-party modules to simplify the process. The first one is the requests module. This is an HTTP library for Python that makes getting HTTP data more convenient than Python's built-in urllib.request and can be installed using python -m pip install requests.

To show its ease of use, we will use Pixabay's API (documented here). Pixabay is a picture website where all images can be reused, making it a very convenient resource. We will focus on fruit pictures. Later on when manipulating the file, we will use the collected fruit pictures, but now we just want to find the fruit pictures.

First, we will quickly see what pictures are available on Pixabay. We will grab a hundred pictures, browse them quickly, and select the one we want. To do this, we need a Pixabay API key, so we need to create an account and then get the key from the Search Image section of the API document.

requests module

The basic versions of using the requests module to make HTTP requests to the API include building HTTP URLs, making requests, and reading responses. Here, the response is in JSON format. The requests module makes each step very simple. The API parameter is a Python dictionary, and the get() function makes a call. If the API returns JSON, the requests will provide it as .json in the response. Therefore, a simple call looks like this:

import requests

PIXABAY_API_KEY = "11111111-7777777777777777777777777"

base_url = "https://pixabay.com/api/"
base_params = {
    "key": PIXABAY_API_KEY,
    "q": "fruit",
    "image_type": "photo",
    "category": "food",
    "safesearch": "true"
}

response = requests.get(base_url, params=base_params)
results = response.json()

This will return a Python object, and as suggested by the API documentation, we can view its various parts:

To get a hundred results, we can simply decide to make five calls, each of which gets 20 results, but this is not robust enough. A better approach is to loop through the request page until you get the desired one hundred results and then stop. This prevents problems when Pixabay changes the default number of results (e.g. to 15). It also allows us to handle the situation where the search terms do not have a hundred pictures. So we use a while loop, incrementing the page number each time, and if we have reached 100 images, or there is no image to retrieve, we exit the loop:

Cache HTTP requests

It is a good idea to avoid making the same requests to the HTTP API multiple times. Many APIs have usage restrictions to avoid overuse by requesters, and requests take time and effort. We should try to avoid duplicating previous requests. Fortunately, there is a useful way to do this when using Python's requests module: Install python -m pip install requests-cache using requests-cache. This will seamlessly record any HTTP calls we make and save the results. Then, later if we make the same call again, we get the locally saved result without having to access the API again. This saves time and bandwidth. To use requests_cache, import it and create a CachedSession and then use session.get instead of requests.get to get the URL, we will get the benefits of caching without extra effort:

Generate output

In order to view the query results, we need to display the picture somewhere. A convenient way is to create a simple HTML page to display each image. Pixabay provides small thumbnails for each image, which is called previewURL in the API response, so we can create an HTML page to display all of these thumbnails and link them to the main Pixabay page - from which we can choose Download the pictures we want and sign the photographer. Therefore, each image in the page might look like this:

We can build it from the images list using list comprehension and then use "n".join() to concatenate all the results into a large string:

Then, if we write out a very simple HTML page with that list, it's easy to open it in a web browser, quickly view all the search results we get from the API and click any of them to jump Download to the full Pixabay page:

This article is excerpted from Practical Python and can be purchased at SitePoint Premium and e-book retailers.

(The following are FAQs, which have been rewritten and streamlined according to the original text)

Frequently Asked Questions about Getting Data with Python's HTTP API (FAQs)

• What is the difference between HTTP and HTTPS? HTTP is a hypertext transfer protocol, and HTTPS is a secure hypertext transfer protocol. The main difference is that HTTPS uses SSL certificates to establish a secure encrypted connection between the server and the client, while HTTP does not. This makes HTTPS more secure when transferring sensitive data such as credit card information or login credentials.

• How does HTTP work in Python? Multiple libraries can be used in Python to issue HTTP requests, the most commonly used is requests. This library allows you to send HTTP requests and process responses, including processing cookies, form data, multi-part files, and more. It is a powerful tool for interacting with web services and can be used in a variety of applications.

• What are the common HTTP methods? How to use them in Python? The most common HTTP methods are GET, POST, PUT, DELETE, HEAD, OPTIONS, and PATCH. In Python, you can use the requests library to use these methods. For example, to send a GET request, you can use requests.get(url), and to send a POST request, you can use requests.post(url, data).

• How to handle HTTP responses in Python? When you use the requests library to send HTTP requests in Python, you get a Response object. This object contains the server's response to your request. If the response is in JSON format, you can use response.text or response.json() to access the content of the response. You can also use response.status_code to check the status code of the response.

• How to use HTTP headers in Python? You can use them in Python by passing HTTP headers as a dictionary to the requests parameter of the headers function. For example, requests.get(url, headers={'User-Agent': 'my-app'}). The header can be used to provide additional information about the request or client, such as user agent, content type, authorization, and so on.

• How to handle cookies in Python? Cookies can be processed in Python using the cookies attribute of the Response object. You can access the cookies sent by the server using response.cookies and send the cookies to the server by passing them as a dictionary to the requests parameter of the cookies function.

• How to send form data using POST request in Python? It can be sent using a POST request in Python by passing the form data as a dictionary to the requests.post parameter of the data function. For example, requests.post(url, data={'key': 'value'}). The requests library will automatically encode the data in the correct format.

• How to send a file using POST request in Python? Files can be sent using POST requests in Python by passing them as dictionary to the requests.post parameter of the files function. The dictionary should contain the name of the file field as the key, and the tuple containing the file name and file object as the values.

• How to deal with errors and exceptions of requests library in Python? The requests library in Python throws exceptions for certain types of errors, such as network errors or timeouts. You can use the try/except block to catch these exceptions and handle them appropriately. You can also check the status code of the response to handle HTTP errors.

• How to make an asynchronous HTTP request in Python? You can use the aiohttp library to issue asynchronous HTTP requests in Python. This library allows you to send HTTP requests asynchronously and process responses, which can significantly improve the performance of your application when handling large numbers of requests.

The above is the detailed content of Fetching Data from an HTTP API with Python. For more information, please follow other related articles on the PHP Chinese website!