How to Build a Product Scraper for Infinite Scroll Websites using ZenRows Web Scraper

In the realm of web scraping, accessing and extracting data from web pages that use infinite scrolling can be a challenge for developers. Many websites use this technique to load more content dynamically, making it hard to scrape all available data in one go. A good solution involves simulating user actions, like clicking a "Load More" button to reveal additional content.
This tutorial will delve into scraping product data from a page with infinite scroll, utilizing Zenrows open-source web scraper, you’ll build a scraper bot that will access contents from a web page, and you’ll use Zenrows to generate more products on the page by clicking the "Load More" button, to simulate an infinite scrolling.

Prerequisites

To follow this tutorial, you need to have the following:

• Python: You should have Python set up on your machine. If not, you can install it here.
• Web Scraping Foundation: You should have a solid grasp of how web scraping works.
• ZenRows SDK: You will be using the ZenRows service to bypass anti-scraping measures and simplify scraping dynamic content. You can sign up for a free ZenRows account here.

Getting Access to the Content

Once you have signed up for your Zenrows account and you have the prerequisites in place, the next step is to access the content from the web page; for this tutorial, you’ll be using this page https://www.scrapingcourse.com/button-click.

You'll also use ZenRows SDK to scrape the dynamic pages and handle various rendering and anti-bot measures. Let’s get you started:

Install the required Libraries:
Open the terminal of your preferred IDE and run the code to install the ZenRows Python SDK.

pip install zenrows python-dotenv

Setting Up your API

Head over to your dashboard and copy the API Key at the top right corner of your screen like in the image below.

Next, create the pages app.py and .env, then add the code below to your app.py file. And add your API key to the variable API_KEY in your .env file.

# Import ZenRows SDK
from zenrows import ZenRowsClient
from dotenv import load_dotenv
import os

# Load environment variables
load_dotenv()

# Initialize ZenRows client with your API key
client = ZenRowsClient(os.getenv("API_KEY"))

# URL of the page you want to scrape
url = "https://www.scrapingcourse.com/button-click"

# Set up initial parameters for JavaScript rendering and interaction
base_params = {
    "js_render": "true",
    "json_response": "true",
    "premium_proxy": "true",
    "markdown_response": "true"
}

The code above initiates the Zenrow SDK using your API key. It sets up the URL variable for the webpage you’ll be scraping and establishes the base_params variable for the necessary parameters. You can execute the scraper using the command:

python app.py

This will provide you with the HTML representation of the page containing only the products on the current page.
.

You can always take this one step further.

Loading More Products

To enhance your scraper, you can implement additional parameters to interact with the "Load More" button at the bottom of the webpage and load more products.

Start by modifying your imports to include the necessary packages and adding a parse_products function that filters the product response:

pip install zenrows python-dotenv

Next, create a while loop to continuously scrape product information from multiple pages until a specified limit (max_products). Set the limit to 50 for this tutorial:

# Import ZenRows SDK
from zenrows import ZenRowsClient
from dotenv import load_dotenv
import os

# Load environment variables
load_dotenv()

# Initialize ZenRows client with your API key
client = ZenRowsClient(os.getenv("API_KEY"))

# URL of the page you want to scrape
url = "https://www.scrapingcourse.com/button-click"

# Set up initial parameters for JavaScript rendering and interaction
base_params = {
    "js_render": "true",
    "json_response": "true",
    "premium_proxy": "true",
    "markdown_response": "true"
}

This loop will continue to scrap products by simulating the clicking of the "Load More" button until the specified limit is reached.
Parsing Product Information
Finally, you can parse the product information you scraped in the previous step. For each product, extract the product name, image link, price, and product page URL. You can also calculate the total price of all products and print the results as follows:

python app.py

Parsing Into a CSV file

If you would prefer to parse your response into an exported csv file, in the next few steps, you'll take the product information you've scraped and learn how to export it to a CSV file.

Modifying the Script to Save Data

First, you need to use Python's built-in CSV module to save the product data. In this case, each product has four main attributes: name, image_link, price, and product_url.

You can use them as the headers for your CSV, loop through the list of scraped products, and then write each product as a row in the CSV file.

import re
import json
import time

def parse_products(response_json):
    try:
        data = json.loads(response_json)
        md_content = data.get('md', '')
        pattern = r'\[!\[([^\]]+)\]\(([^\)]+)\)\*\n([^\\n]+)\*\n\*\n$(\d+)\]\(([^\)]+)\)'
        matches = re.findall(pattern, md_content)

        products = []
        for match in matches:
            product = {
                'name': match[0],
                'image_link': match[1],
                'price': int(match[3]),
                'product_url': match[4]
            }
            products.append(product)

        return products
    except json.JSONDecodeError:
        print("Error: Unable to parse JSON response")
        print("Response content:", response_json[:500])
        return []
    except Exception as e:
        print(f"Error parsing products: {str(e)}")
        return []

# Zenrow SDK code here

Now, after scraping the data, just call the save_to_csv(all_products) function to store the data in a CSV file named products.csv.

Run the command to automatically save the data to a CSV file once the scraping process is complete.

# Zenrow SDK code goes here

max_products = 50
all_products = []
page = 1

while len(all_products) < max_products:
    print(f"Scraping page {page}...")

    # Update parameters for each request
    params = base_params.copy()
    js_instructions = [{"click": "#load-more-btn"} for _ in range(page)]
    js_instructions.append({"wait": 5000})
    params["js_instructions"] = json.dumps(js_instructions)

    try:
        # Send the GET request to ZenRows
        response = client.get(url, params=params)

        # Parse the response JSON
        new_products = parse_products(response.text)

        if not new_products:
            print("No more products found. Stopping.")
            break

        all_products.extend(new_products)
        print(f"Found {len(new_products)} products on this page.")
        print(f"Total products so far: {len(all_products)}")

        page += 1

        # Add a delay to avoid overwhelming the server
        time.sleep(2)

    except Exception as e:
        print(f"Error occurred: {str(e)}")
        break

Identifying the 5 Highest-Priced Products

Now that you have all the products in a structured format, you can go a step further and identify the 5 highest-priced products, and you’ll have to visit each product page to extract extra details like the product description and SKU code.

Sorting the Products by Price: Using Python's sorted() function, you can sort the product list by the price in descending order and retrieve the top 5 products.

You’ll need to visit each page using the requests.get() function to fetch the product data for each of them. From the response, you can extract the product description and SKU code.
You can also update the csv flie from the last step to include the additional details.

Here is the code to achieve that:

# Updated Params and while loop code goes here
# Calculate the total price of all products
total_sum = sum(product['price'] for product in all_products)

print("\nAll products:")
for product in all_products:
    print(product)

# Print the total sum of the product prices
print(f"\nTotal number of products: {len(all_products)}")
print(f"Total sum of product prices: ${total_sum}")

Now, after scraping, you can now identify the highest-priced products:

pip install zenrows python-dotenv

After retrieving the additional information, you can either modify the CSV file or create a new one with these details included.

The Complete Code

Here is how your complete app.py file should look like.

# Import ZenRows SDK
from zenrows import ZenRowsClient
from dotenv import load_dotenv
import os

# Load environment variables
load_dotenv()

# Initialize ZenRows client with your API key
client = ZenRowsClient(os.getenv("API_KEY"))

# URL of the page you want to scrape
url = "https://www.scrapingcourse.com/button-click"

# Set up initial parameters for JavaScript rendering and interaction
base_params = {
    "js_render": "true",
    "json_response": "true",
    "premium_proxy": "true",
    "markdown_response": "true"
}

Here is what a successful response would look like.

python app.py

Check out the complete codebase on GitHub.

Conclusion

In this tutorial, you learned how to scrape products from a webpage with infinite scrolling using the "Load More" button. By following the outlined steps, you can extract valuable product information and enhance your scraping techniques using ZenRows.
To know more about how you can use Zenrow web scraping tools, check out the following articles on our blog.

• How to parse HTML with PHP
• How to use Hrequests for Web Scraping
• How to use playwright in Ruby Here’s a quick video on a no-code approach to using Zenrows Web scraping tools.

The above is the detailed content of How to Build a Product Scraper for Infinite Scroll Websites using ZenRows Web Scraper. For more information, please follow other related articles on the PHP Chinese website!