Asynchronous HTTP Requests in Python with HTTPX and asyncio

Asynchronous programming is increasingly vital in Python development. With asyncio now a standard library component and many compatible third-party packages, this paradigm is here to stay. This tutorial demonstrates using the HTTPX library for asynchronous HTTP requests—a prime use case for non-blocking code.

What is Non-Blocking Code?

Terms like "asynchronous," "non-blocking," and "concurrent" can be confusing. Essentially:

• Asynchronous routines can "pause" while awaiting results, allowing other routines to execute concurrently.
• This creates the appearance of concurrent execution, even though true parallelism may not be involved.

Asynchronous code avoids blocking, enabling other code to run while waiting for results. The asyncio library provides tools for this, and aiohttp offers specialized HTTP request functionality. HTTP requests are ideal for asynchronicity because they involve waiting for server responses, a period where other tasks can efficiently execute.

Setup

Ensure your Python environment is configured. Refer to a virtual environment guide if needed (Python 3.7 is required). Install HTTPX:

pip install httpx==0.18.2

Making an HTTP Request with HTTPX

This example uses a single GET request to the Pokémon API to fetch data for Mew (Pokémon #151):

import asyncio
import httpx

async def main():
    url = 'https://pokeapi.co/api/v2/pokemon/151'
    async with httpx.AsyncClient() as client:
        response = await client.get(url)
        pokemon = response.json()
        print(pokemon['name'])

asyncio.run(main())

async designates a coroutine; await yields control to the event loop, resuming execution upon result availability.

Making Multiple Requests

The real power of asynchronicity is evident when making numerous requests. This example fetches data for the first 150 Pokémon:

import asyncio
import httpx
import time

start_time = time.time()

async def main():
    async with httpx.AsyncClient() as client:
        for number in range(1, 151):
            url = f'https://pokeapi.co/api/v2/pokemon/{number}'
            response = await client.get(url)
            pokemon = response.json()
            print(pokemon['name'])

asyncio.run(main())
print(f"--- {time.time() - start_time:.2f} seconds ---")

Time the execution. Compare this with a synchronous approach.

Synchronous Request Comparison

The synchronous equivalent:

import httpx
import time

start_time = time.time()
client = httpx.Client()
for number in range(1, 151):
    url = f'https://pokeapi.co/api/v2/pokemon/{number}'
    response = client.get(url)
    pokemon = response.json()
    print(pokemon['name'])

print(f"--- {time.time() - start_time:.2f} seconds ---")

Note the runtime difference. HTTPX's connection pooling minimizes the disparity, but asyncio offers further optimization.

Advanced Asynchronous Techniques

For superior performance, run requests concurrently using asyncio.ensure_future and asyncio.gather:

import asyncio
import httpx
import time

start_time = time.time()

async def fetch_pokemon(client, url):
    response = await client.get(url)
    return response.json()['name']

async def main():
    async with httpx.AsyncClient() as client:
        tasks = [asyncio.ensure_future(fetch_pokemon(client, f'https://pokeapi.co/api/v2/pokemon/{number}')) for number in range(1, 151)]
        pokemon_names = await asyncio.gather(*tasks)
        for name in pokemon_names:
            print(name)

asyncio.run(main())
print(f"--- {time.time() - start_time:.2f} seconds ---")

This significantly reduces execution time by running requests concurrently. The total time approaches the duration of the longest single request.

Conclusion

Using HTTPX and asynchronous programming dramatically improves performance for multiple HTTP requests. This tutorial provides a basic introduction to asyncio; explore its capabilities further to enhance your Python projects. Consider exploring aiohttp for alternative asynchronous HTTP request handling.

The above is the detailed content of Asynchronous HTTP Requests in Python with HTTPX and asyncio. For more information, please follow other related articles on the PHP Chinese website!