search
HomeBackend DevelopmentPython TutorialDetailed explanation of scrapy examples of python crawler framework

Generate Project

Scrapy provides a tool to generate projects. Some files are preset in the generated project, and users need to add their own code to these files.

Open the command line and execute: scrapy startproject tutorial. The generated project has a structure similar to the following

tutorial/

  scrapy.cfg

  tutorial/

    __init__.py

  items.py

 pipelines.py

settings .py

            spiders/

                                                                                                                                                           The name attribute is important , different spiders cannot use the same name

start_urls is the starting point for spiders to crawl web pages, and can include multiple URLs

parse method is the callback called by default after spider captures a web page, avoid using this name to define your own method .

When the spider gets the content of the url, it will call the parse method and pass it a response parameter. The response contains the content of the captured web page. In the parse method, you can parse the data from the captured web page. The code above simply saves the web page content to a file.

Start crawling

You can open the command line, enter the generated project root directory tutorial/, and execute scrapy crawl dmoz, where dmoz is the name of the spider.


Parse web page content

scrapy provides a convenient way to parse data from web pages, which requires the use of HtmlXPathSelector

from scrapy.spider import BaseSpider
class DmozSpider(BaseSpider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]
    def parse(self, response):
        filename = response.url.split("/")[-2]
        open(filename, 'wb').write(response.body)

HtmlXPathSelector uses Xpath to parse data


//ul/li means to select all ul tags The li tag below

a/@href means selecting the href attribute of all a tags

a/text() means selecting the a tag text

a[@href="abc"] means selecting all a whose href attribute is abc Tag

We can save the parsed data in an object that scrapy can use, and then scrapy can help us save these objects without having to save the data to a file ourselves. We need to add some classes to items.py, which are used to describe the data we want to save

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
class DmozSpider(BaseSpider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]
    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        sites = hxs.select('//ul/li')
        for site in sites:
            title = site.select('a/text()').extract()
            link = site.select('a/@href').extract()
            desc = site.select('text()').extract()
            print title, link, desc

When executing scrapy on the command line, we can add two parameters to let scrapy output the items returned by the parse method to json In the file

scrapy crawl dmoz -o items.json -t json

items.json will be placed in the root directory of the project

Let scrapy automatically crawl all links on the webpage

In the example above, scrapy Only the contents of the two URLs in start_urls are crawled, but usually what we want to achieve is for scrapy to automatically discover all the links on a web page, and then crawl the contents of these links. In order to achieve this, we can extract the links we need in the parse method, then construct some Request objects and return them. Scrapy will automatically crawl these links. The code is similar:

from scrapy.item import Item, Field
class DmozItem(Item):
   title = Field()
   link = Field()
   desc = Field()
然后在spider的parse方法中,我们把解析出来的数据保存在DomzItem对象中。
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from tutorial.items import DmozItem
class DmozSpider(BaseSpider):
   name = "dmoz"
   allowed_domains = ["dmoz.org"]
   start_urls = [
       "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
       "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
   ]
   def parse(self, response):
       hxs = HtmlXPathSelector(response)
       sites = hxs.select('//ul/li')
       items = []
       for site in sites:
           item = DmozItem()
           item['title'] = site.select('a/text()').extract()
           item['link'] = site.select('a/@href').extract()
           item['desc'] = site.select('text()').extract()
           items.append(item)
       return items

parse is the default callback, which returns a Request list. Scrapy automatically crawls web pages based on this list. Whenever a web page is captured, parse_item will be called, and parse_item will also return a list. Scrapy will The web page will be crawled based on this list, and parse_details will be called after crawling


In order to make such work easier, scrapy provides another spider base class, using which we can easily implement automatic crawling of links. We need to use CrawlSpider

class MySpider(BaseSpider):
    name = 'myspider'
    start_urls = (
        'http://example.com/page1',
        'http://example.com/page2',
        )
    def parse(self, response):
        # collect `item_urls`
        for item_url in item_urls:
            yield Request(url=item_url, callback=self.parse_item)
    def parse_item(self, response):
        item = MyItem()
        # populate `item` fields
        yield Request(url=item_details_url, meta={'item': item},
            callback=self.parse_details)
    def parse_details(self, response):
        item = response.meta['item']
        # populate more `item` fields
        return item

Compared with BaseSpider, the new class has an additional rules attribute. This attribute is a list, which can contain multiple Rules. Each Rule describes which links need to be crawled and which do not. This is the documentation for the Rule class http://doc.scrapy.org/en/latest/topics/spiders.html#scrapy.contrib.spiders.Rule

These rules can have callbacks or not, when there is no callback , scrapy simply follows all these links.

Usage of pipelines.py

In pipelines.py we can add some classes to filter out the items we don’t want and save the items to the database.

from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
class MininovaSpider(CrawlSpider):
    name = 'mininova.org'
    allowed_domains = ['mininova.org']
    start_urls = ['http://www.mininova.org/today']
    rules = [Rule(SgmlLinkExtractor(allow=['/tor/\d+'])),
             Rule(SgmlLinkExtractor(allow=['/abc/\d+']), 'parse_torrent')]
    def parse_torrent(self, response):
        x = HtmlXPathSelector(response)
        torrent = TorrentItem()
        torrent['url'] = response.url
        torrent['name'] = x.select("//h1/text()").extract()
        torrent['description'] = x.select("//div[@id='description']").extract()
        torrent['size'] = x.select("//div[@id='info-left']/p[2]/text()[2]").extract()
        return torrent

If the item does not meet the requirements, then an exception will be thrown and the item will not be output to the json file.

To use pipelines, we also need to modify settings.py

Add a line

ITEM_PIPELINES = ['dirbot.pipelines.FilterWordsPipeline']

Now execute scrapy crawl dmoz -o items.json -t json, which does not meet the requirements The item was filtered out

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
What data types can be stored in a Python array?What data types can be stored in a Python array?Apr 27, 2025 am 12:11 AM

Pythonlistscanstoreanydatatype,arraymodulearraysstoreonetype,andNumPyarraysarefornumericalcomputations.1)Listsareversatilebutlessmemory-efficient.2)Arraymodulearraysarememory-efficientforhomogeneousdata.3)NumPyarraysareoptimizedforperformanceinscient

What happens if you try to store a value of the wrong data type in a Python array?What happens if you try to store a value of the wrong data type in a Python array?Apr 27, 2025 am 12:10 AM

WhenyouattempttostoreavalueofthewrongdatatypeinaPythonarray,you'llencounteraTypeError.Thisisduetothearraymodule'sstricttypeenforcement,whichrequiresallelementstobeofthesametypeasspecifiedbythetypecode.Forperformancereasons,arraysaremoreefficientthanl

Which is part of the Python standard library: lists or arrays?Which is part of the Python standard library: lists or arrays?Apr 27, 2025 am 12:03 AM

Pythonlistsarepartofthestandardlibrary,whilearraysarenot.Listsarebuilt-in,versatile,andusedforstoringcollections,whereasarraysareprovidedbythearraymoduleandlesscommonlyusedduetolimitedfunctionality.

What should you check if the script executes with the wrong Python version?What should you check if the script executes with the wrong Python version?Apr 27, 2025 am 12:01 AM

ThescriptisrunningwiththewrongPythonversionduetoincorrectdefaultinterpretersettings.Tofixthis:1)CheckthedefaultPythonversionusingpython--versionorpython3--version.2)Usevirtualenvironmentsbycreatingonewithpython3.9-mvenvmyenv,activatingit,andverifying

What are some common operations that can be performed on Python arrays?What are some common operations that can be performed on Python arrays?Apr 26, 2025 am 12:22 AM

Pythonarrayssupportvariousoperations:1)Slicingextractssubsets,2)Appending/Extendingaddselements,3)Insertingplaceselementsatspecificpositions,4)Removingdeleteselements,5)Sorting/Reversingchangesorder,and6)Listcomprehensionscreatenewlistsbasedonexistin

In what types of applications are NumPy arrays commonly used?In what types of applications are NumPy arrays commonly used?Apr 26, 2025 am 12:13 AM

NumPyarraysareessentialforapplicationsrequiringefficientnumericalcomputationsanddatamanipulation.Theyarecrucialindatascience,machinelearning,physics,engineering,andfinanceduetotheirabilitytohandlelarge-scaledataefficiently.Forexample,infinancialanaly

When would you choose to use an array over a list in Python?When would you choose to use an array over a list in Python?Apr 26, 2025 am 12:12 AM

Useanarray.arrayoveralistinPythonwhendealingwithhomogeneousdata,performance-criticalcode,orinterfacingwithCcode.1)HomogeneousData:Arrayssavememorywithtypedelements.2)Performance-CriticalCode:Arraysofferbetterperformancefornumericaloperations.3)Interf

Are all list operations supported by arrays, and vice versa? Why or why not?Are all list operations supported by arrays, and vice versa? Why or why not?Apr 26, 2025 am 12:05 AM

No,notalllistoperationsaresupportedbyarrays,andviceversa.1)Arraysdonotsupportdynamicoperationslikeappendorinsertwithoutresizing,whichimpactsperformance.2)Listsdonotguaranteeconstanttimecomplexityfordirectaccesslikearraysdo.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

ZendStudio 13.5.1 Mac

ZendStudio 13.5.1 Mac

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

MantisBT

MantisBT

Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.