What is needed for Python crawler to crawl images?

What does a Python crawler need to crawl images? Below are two methods to make batch crawling of network images:

The first method: based on urllib implementation

The main points are as follows:

1.url_request = request.Request(url)

2.url_response = request.urlopen(url) or url_response = request.urlopen(url_request)

3.data= url_response .read().decode('utf-8')

4.jpglist=re.findall(regular expression, data)

5.request.urlretrieve(jpgUrl,'% s.jpg' %n) #Download, the first parameter URL, the second parameter name

Related recommendations: "Python Video Tutorial"

The first case , we crawled images on a webpage on Maopu. The case code is as follows.

It should be noted that "pic2\\" in the code request.urlretrieve(each,'pic2\\%s.jpg' %n) means that the downloaded picture is placed in In the folder pic2 created in advance. After running the code, IDLE and the final result are shown in the figure below.

The second implementation method: based on requests

The key points are as follows:

1.data=requests .get(url).text

2.jpglist=re.findall(regular expression,data,re.S)

3.pic=requests.get(pic_url,timeout=10 )

4. fp=open(pic_name,'wb')

fp.write(pic.content)

fp.close()

In this case, we crawled the webpage of a wallpaper website, and we predict that the image quality should be relatively high. The complete code is as follows:

# Let’s take a look at the results of running the code, as shown in the figure below. It's really nice, we got 42 wallpapers and it was done in just a few seconds.

The above is the detailed content of What is needed for Python crawler to crawl images?. For more information, please follow other related articles on the PHP Chinese website!