Python crawler: crawl Baidu images through keywords

Tools used: Python2.7

scrapy framework

sublime text3

One. Build python (Windows version)

1. Install python2.7 ---Then enter python in cmd. If the interface is as follows, the installation is successful

2. Integrate the Scrapy framework----Enter the command line: pip install Scrapy

The successful installation interface is as follows:

Failed There are many situations, here is an example:

Solution:

You can search Baidu for other errors.

two. Start programming.

The python code is as follows:

Code comments: Two modules urllib,re are introduced. Define two functions. The first function is to obtain the entire target web page data. The second function is to obtain the target image in the target web page, traverse the web page, and sort the acquired images starting from 0.

Note: re module knowledge points:

Crawling picture renderings:

pictures The saving path defaults to the same directory as the created .py file.

2. Crawling Baidu images with anti-crawler measures. Such as Baidu pictures, etc.

For example, the keyword search "emoticon package" https://image.baidu.com/search/index?tn=baiduimage&ct=201326592&lm=-1&cl=2&ie=gbk&word=%B1%ED%C7%E9% B0%FC&fr=ala&ori_query=%E8%A1%A8%E6%83%85%E5%8C%85&ala=0&alatpl=sp&pos=0&hs=2&xthttps=111111

The pictures are loaded in a scrolling manner, and the most pictures are crawled first 30 priority photos.

The code is as follows:

Code comments: Import 4 modules, and the os module is used to specify the save path. The first two functions are the same as above. The third function uses an if statement and a tryException.

The crawling process is as follows:

Crawling results:

Note: Write python code Pay attention to alignment, and do not mix tabs and spaces, as it is easy to report errors.

For more Python crawlers: Crawling Baidu images through keywords, please pay attention to the PHP Chinese website for related articles!