Home > Article > Backend Development > Distributed crawlers in Scrapy and methods to improve data crawling efficiency
Scrapy is an efficient Python web crawler framework that can write crawler programs quickly and flexibly. However, when processing large amounts of data or complex websites, stand-alone crawlers may encounter performance and scalability issues. At this time, distributed crawlers need to be used to improve data crawling efficiency. This article introduces distributed crawlers in Scrapy and methods to improve data crawling efficiency.
1. What is a distributed crawler?
In the traditional stand-alone crawler architecture, all crawlers run on the same machine. When faced with large amounts of data or high-pressure crawling tasks, machine performance is often tight. Distributed crawlers distribute crawler tasks to multiple machines for processing. Through distributed computing and storage, the burden on a single machine is reduced, thereby improving the efficiency and stability of the crawler.
Distributed crawlers in Scrapy are usually implemented using the open source distributed scheduling framework Distributed Scrapy (DSC for short). DSC distributes Scrapy crawler programs to multiple machines for parallel processing, and uniformly summarizes the results to the central scheduling node.
2. How to implement distributed crawler?
1. Install Distributed Scrapy
Run the following command to install DSC:
pip install scrapy_redis
pip install pymongo
2. Modify Scrapy configuration file
Add the following configuration in the settings.py file of the Scrapy project:
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
SCHEDULER_PERSIST=True
REDIS_HOST='localhost'
REDIS_PORT=6379
3. Write the crawler code
In the Scrapy crawler program , you need to modify the starting request method, use the starting method of scrapy-redis:
import scrapy,re,json
from ..items import DouyuItem
from scrapy_redis.spiders import RedisSpider
class DouyuSpider(RedisSpider):
# 爬虫名字 name = 'douyu' # redis-key,从redis中pop数据进行爬取 redis_key = 'douyu:start_urls' def parse(self, response): # scrapy爬虫代码
4. Start the redis service
Execute the following command in the terminal to start the redis service :
redis-server
5. Start Distributed Scrapy
Enter the following command in the terminal to start the DSC node:
scrapy crawl douyu -s JOBDIR= job1
Among them, job1 can be a custom name, which is used for DSC to record the crawler status.
3. Optimize Scrapy crawler
Scrapy provides many methods to optimize crawler efficiency. If used with distributed crawlers, data crawling efficiency can be further improved.
1. Using CrawlerRunner
CrawlerRunner requires a Twisted class to extend the application. Compared to simply running a Python file, it allows you to run multiple crawlers simultaneously in the same process without using multiple processes or multiple machines. This can make task management easier.
The way to use CrawlerRunner is as follows:
from twisted.internet import reactor,defer
from scrapy.crawler import CrawlerRunner
from scrapy.utils.project import get_project_settings
from my_spider.spiders.my_spider import MySpider
runner = CrawlerRunner(get_project_settings())
@defer.inlineCallbacks
def crawl():
yield runner.crawl(MySpider) reactor.stop()
crawl()
reactor.run()
2. Reduce the priority of the download middleware
If you need to process a large amount or complex data in the download middleware, you can use CONCURRENT_REQUESTS_PER_DOMAIN to reduce the priority of the download middleware. Priority:
CONCURRENT_REQUESTS_PER_DOMAIN = 2
DOWNLOAD_DELAY = 0.5
DOWNLOADER_MIDDLEWARES = {
'myproject.middlewares.MyCustomDownloaderMiddleware': 543,
}
3. Adjustment The CONCURRENT_REQUESTS and DOWNLOAD_DELAY parameters
CONCURRENT_REQUESTS indicate the maximum number of requests that each domain name can handle simultaneously, and can be reasonably adjusted according to machine configuration and task requirements.
DOWNLOAD_DELAY represents the delay time between each request. The crawler efficiency can be improved by increasing the delay or asynchronous requests.
4. Summary
Scrapy’s distributed crawler can help us quickly process large amounts of data and improve crawler efficiency. At the same time, crawler efficiency can be further improved by lowering the priority of the download middleware, adjusting the number of coroutines, and increasing the request delay. Distributed crawler is one of the important functions of Scrapy. Learning it can allow us to easily handle various crawler tasks.
The above is the detailed content of Distributed crawlers in Scrapy and methods to improve data crawling efficiency. For more information, please follow other related articles on the PHP Chinese website!