The simplest web crawler tutorial in python

When we browse the Internet every day, we often see some good-looking pictures, and we want to save and download these pictures, or use them as desktop wallpapers, or as design materials. The following article will introduce to you the relevant information about using python to implement the simplest web crawler. Friends in need can refer to it. Let's take a look together.

Preface

Web crawlers (also known as web spiders, web robots, among the FOAF community, more often called web crawlers ) is a program or script that automatically captures World Wide Web information according to certain rules. Recently, I have become very interested in python crawlers. I would like to share my learning path here and welcome your suggestions. We communicate with each other and make progress together. Not much to say, let’s take a look at the detailed introduction:

1. Development tools

The author uses The best tool is sublime text3. Its short and concise (maybe men don't like this word) fascinates me very much. It is recommended for everyone to use. Of course, if your computer configuration is good, pycharm may be more suitable for you.

sublime text3 To build a python development environment, it is recommended to view this article:

[sublime to build a python development environment]

As the name suggests, crawlers are like bugs , crawling on the big web of the Internet. In this way, we can get what we want.

Since we want to crawl on the Internet, we need to understand the URL, the legal name "Uniform Resource Locator", and the nickname "Link". Its structure mainly consists of three parts:

(1) Protocol: such as the HTTP protocol we commonly see in URLs.

(2) Domain name or IP address: Domain name, such as: www.baidu.com, IP address, that is, the corresponding IP after domain name resolution.

(3) Path: directory or file, etc.

3. urllib develops the simplest crawler

(1) Introduction to urllib

##urllib.errorException classes raised by urllib.request.urllib.parseParse URLs into or assemble them from components.urllib.requestExtensible library for opening URLs.urllib.responseResponse classes used by urllib.urllib.robotparserLoad a robots.txt file and answer questions about fetchability of other URLs.

Module	Introduce

（2 ) Develop the simplest crawler

The Baidu homepage is simple and elegant, which is very suitable for our crawlers.

The crawler code is as follows:

from urllib import request

def visit_baidu():
 URL = "http://www.baidu.com"
 # open the URL
 req = request.urlopen(URL)
 # read the URL 
 html = req.read()
 # decode the URL to utf-8
 html = html.decode("utf_8")
 print(html)

if __name__ == '__main__':
 visit_baidu()

The result is as shown below:

We can compare with our running results by right-clicking on the blank space of Baidu homepage and viewing the review elements.

Of course, request can also generate a request object, which can be opened using the urlopen method.

The code is as follows:

from urllib import request

def vists_baidu():
 # create a request obkect
 req = request.Request('http://www.baidu.com')
 # open the request object
 response = request.urlopen(req)
 # read the response 
 html = response.read()
 html = html.decode('utf-8')
 print(html)

if __name__ == '__main__':
 vists_baidu()

The running result is the same as before.

(3) Error handling

Error handling is handled through the urllib module, mainly including URLError and HTTPError errors, of which HTTPError error is URLError error Subclasses of HTTRPError can also be caught by URLError.

HTTPError can be captured through its code attribute.

The code for handling HTTPError is as follows:

##

from urllib import request
from urllib import error

def Err():
 url = "https://segmentfault.com/zzz"
 req = request.Request(url)

 try:
 response = request.urlopen(req)
 html = response.read().decode("utf-8")
 print(html)
 except error.HTTPError as e:
 print(e.code)
if __name__ == '__main__':
 Err()

The running result is as shown in the figure:

404 is the printed error code. You can Baidu for detailed information about this.

URLError can be captured through its reason attribute.

chuliHTTPError code is as follows:

##

from urllib import request
from urllib import error

def Err():
 url = "https://segmentf.com/"
 req = request.Request(url)

 try:
 response = request.urlopen(req)
 html = response.read().decode("utf-8")
 print(html)
 except error.URLError as e:
 print(e.reason)
if __name__ == '__main__':
 Err()

The running result is as shown in the figure:

Since in order to handle errors, it is best to write both errors into the code. After all, the more detailed the code, the clearer it will be. It should be noted that HTTPError is a subclass of URLError, so HTTPError must be placed in front of URLError, otherwise URLError will be output, such as 404 as Not Found.

The code is as follows:

from urllib import request
from urllib import error

# 第一种方法，URLErroe和HTTPError
def Err():
 url = "https://segmentfault.com/zzz"
 req = request.Request(url)

 try:
 response = request.urlopen(req)
 html = response.read().decode("utf-8")
 print(html)
 except error.HTTPError as e:
 print(e.code)
 except error.URLError as e:
 print(e.reason)

You can change the url to view the output forms of various errors.

Summarize

The above is the detailed content of The simplest web crawler tutorial in python. For more information, please follow other related articles on the PHP Chinese website!