A must-read for new crawlers: Scrapy Getting Started Guide

In terms of data acquisition, Web crawlers have become an indispensable tool. However, for those new to learning and mastering web scraping techniques, choosing the right tools and frameworks can be confusing. Among the many web crawling tools, Scrapy is a very popular tool. Scrapy is an open source Python framework that provides a flexible approach to processing and extracting data.

In this article, I will introduce you to the basics of Scrapy and introduce how to build a simple web crawler in Scrapy.

1. Scrapy Getting Started Guide

1. Installing Scrapy

Before you begin, you first need to install Scrapy. The installation of Scrapy is very simple, just execute the following command in the command line:

pip install scrapy

2. Creating a Scrapy project

When creating a Scrapy project, you can use the following command:

scrapy startproject <project_name>

This will create a folder named 4fc59f85b86f3cdcf539b23b3b2209e3 in the current directory and create the required files and folders in it.

3. Create Spider

In Scrapy, Spider is the main component we use to crawl data. Spider defines how to start requesting URLs, how to follow links, and how to parse pages. In Scrapy, we can use the following command to create a Spider:

scrapy genspider <spider_name> <domain_name>

This will create a new Spider in the project and save it in the spiders directory. You can define the request and parsing methods we need by editing the Spider.

4. Configuring the website to be crawled

It is very important to configure the website to be crawled. We need to define the website URL to be crawled in the Spider file, and how to configure the request. In Scrapy, this function can be achieved by writing the start_requests method. This method will be called when the Spider starts and sends a request from a specific URL.

5. Page parsing

In Scrapy, parsing web pages is the most important step. We can use XPath or CSS selectors to parse the page to extract the required data. In Spider code, you can parse the page by writing the parse method and using the above tools.

6. Storing Data

Finally, we need to store the extracted data in a database or file. In Scrapy, you can use Pipeline to achieve this operation. Pipeline is a mechanism for processing data. It defines specific methods for data cleaning, filtering, transformation, storage, output, etc.

2. A simple example

Next, let’s write a simple Spider and use Scrapy to grab the data of the Top 250 Douban movies. First, create a new project using the following command in the command line:

scrapy startproject tutorial

Go into the tutorial folder and create a Spider named douban_spider:

scrapy genspider douban_spider movie.douban.com

Next, we need to configure the Spider to Request the page and parse the web page. Add the following code to the Spider file:

import scrapy

class DoubanSpider(scrapy.Spider):
    name = "douban"
    allowed_domains = ["movie.douban.com"]
    start_urls = [
        "https://movie.douban.com/top250"
    ]

    def parse(self, response):
        for sel in response.xpath('//div[@class="info"]'):
            title = sel.xpath('div[@class="hd"]/a/span/text()').extract()
            yield {'title': title}

In the above code, we first define the name of the Spider and the domain name of the crawled website. Next, we defined the URLs we wanted to crawl and wrote the parse method to parse the page and extract the data we needed.

For each element with a class attribute of "info", we use XPath to extract the elements containing the movie title and return these elements using the yield keyword.

Finally, we need to save the extracted data. A new Pipeline can be created to process and store the extracted data. The following is a simple Pipeline that saves the extracted data in a JSON file:

import json

class TutorialPipeline(object):

    def __init__(self):
        self.file = open('douban_top250.json', 'w')

    def process_item(self, item, spider):
        line = json.dumps(dict(item)) + "
"
        self.file.write(line)
        return item

    def spider_closed(self, spider):
        self.file.close()

Finally, we need to configure the Pipeline in settings.py. Just add the following code in ITEM_PIPELINES:

ITEM_PIPELINES = {
    'tutorial.pipelines.TutorialPipeline': 100,
}

Now, we have written a simple Scrapy Spider and can start it by executing the following command:

scrapy crawl douban

Execute the After the command, Scrapy will start requesting the page and parsing the data. The extracted data will be saved in a JSON file.

3. Conclusion

Scrapy is a very flexible and powerful web crawler framework. With Scrapy, we can easily build an efficient and scalable web crawler and extract the required data. This article introduces the basics of Scrapy and provides a simple example, hoping to help novices who are learning web crawlers.

The above is the detailed content of A must-read for new crawlers: Scrapy Getting Started Guide. For more information, please follow other related articles on the PHP Chinese website!