Why do I get the 'list out of range' error when using Python crawler?

"list index out of range" error in Python crawler: Cause and solution

When using Python and BeautifulSoup for web crawling, you often encounter list index out of range errors. This problem can occur even if the code is not modified, especially when dealing with dynamic web pages or website structure changes. This article analyzes the cause of this error and provides an effective solution.

Here is a sample code that demonstrates what might cause this error to occur:

 import requests
from bs4 import BeautifulSoup

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.0.0'}
response = requests.get("https://www.iqiyi.com/ranks1/3/0", headers=headers)
print(response.status_code)
response = response.text
soup = BeautifulSoup(response, "html.parser")

def extract_data():
    titles = [title.get_text().strip() for title in soup.find_all("div", class_="rvi__tit1")]
    heat = [heat.get_text().strip() for heat in soup.find_all("span", class_="rvi__index__num")]
    introductions = [intro.get_text().strip() for intro in soup.find_all("p", class_="rvi__des2")]
    return titles, heat, introductions

def display_data(titles, heat, introductions):
    min_len = min(len(titles), len(heat), len(introductions))
    for i in range(min_len):
        print(f"Ranking: {i 1}, Title: {titles[i]}, Popularity: {heat[i]}, Introduction: {introductions[i]}")


if __name__ == '__main__':
    titles, heat, introductions = extract_data()
    display_data(titles, heat, introductions)

In this example, list index out of range error usually occurs in display_data function. The reason is: the lengths of the three lists of titles , heat , and introductions may be inconsistent. If one of the lists has a length less than 10 (or the range of loops), an index out-of-bounds error will occur when accessing the list elements.

Solution:

The key is to make sure that before accessing the list element, the length of the list is checked and only elements within the valid index range are accessed. The improved code is as follows:

 import requests
from bs4 import BeautifulSoup

# ... (headers and request remains the same) ...

def extract_data():
    # ... (extraction remains the same) ...

def display_data(titles, heat, introductions):
    min_len = min(len(titles), len(heat), len(introductions)) # Find the shortest list
    for i in range(min_len):
        print(f"Ranking: {i 1}, Title: {titles[i]}, Popularity: {heat[i]}, Introduction: {introductions[i]}")


if __name__ == '__main__':
    titles, heat, introductions = extract_data()
    display_data(titles, heat, introductions)

By calculating the shortest length of the three lists min_len and using min_len as the range of the loop, we ensure that no elements outside the list index range are accessed, effectively avoiding list index out of range errors. This is a more robust way of processing that can adapt to changes in different web page structures and data volumes. In addition, adding error handling mechanisms (such as try-except blocks) is also a good programming practice that can handle more complex situations.

The above is the detailed content of Why do I get the 'list out of range' error when using Python crawler?. For more information, please follow other related articles on the PHP Chinese website!