Home >Backend Development >Python Tutorial >Use Python's Scrapy framework to crawl beautiful women's pictures in ten minutes

Use Python's Scrapy framework to crawl beautiful women's pictures in ten minutes

高洛峰
高洛峰Original
2017-02-11 13:30:161978browse

Scrapy is written in Python, lightweight, simple and lightweight, and very convenient to use. Using Scrapy can easily complete the collection of online data. It has completed a lot of work for us without having to expend great efforts to develop it ourselves. This article introduces the use of Python's Scrapy framework to crawl beautiful pictures in ten minutes. Friends who need it can refer to it

Introduction

scrapy is a python The following is a crawler framework that is rich in functions and quick and easy to use. You can use scrapy to quickly develop a simple crawler. A simple example given by the official is enough to prove its power:

Use Pythons Scrapy framework to crawl beautiful womens pictures in ten minutes

rapid development

The 10-minute countdown starts below:

Of course, before starting, you can take a look at the scrapy introductory article we wrote before "Writing Python Crawlers from Zero Basics: Using the Scrapy Framework to Write Crawlers

1. Initialize the project

scrapy startproject mzt
cd mzt
scrapy genspider meizitu meizitu.com

##2. Add spider code:

Define scrapy.Item, add image_urls and images, and prepare for downloading images.

Modify start_urls to the initial page, add parse to process the list page, and add parse_item to process the item page.

Use Pythons Scrapy framework to crawl beautiful womens pictures in ten minutes

3. Modify the configuration file:

DOWNLOAD_DELAY = 1 # 添加下载延迟配置
ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1} # 添加图片下载 pipeline
IMAGES_STORE = '.' # 设置图片保存目录

4 . Run the project:

scrapy crawl meizitu

Look at the project operation renderings

Use Pythons Scrapy framework to crawl beautiful womens pictures in ten minutes

Wait for a while, it’s time to harvest

Use Pythons Scrapy framework to crawl beautiful womens pictures in ten minutes

Use Pythons Scrapy framework to crawl beautiful womens pictures in ten minutes##More uses Python’s Scrapy framework to crawl beautiful women in ten minutes For pictures related articles, please pay attention to the PHP Chinese website!

Statement:
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn