[Python] Web Crawler (1): The meaning of crawling web pages and the basic composition of URLs

1. The definition of web crawler

Web crawler, or Web Spider, is a very vivid name.

If the Internet is compared to a spider web, then a spider is a spider crawling around on the web.
Web spiders search for web pages through their link addresses.

Start from a certain page of the website (usually the homepage), read the content of the webpage, find other link addresses in the webpage,

and then use these link addresses to find the next webpage, so The loop continues until all web pages of this website have been crawled.

If the entire Internet is regarded as a website, then web spiders can use this principle to crawl all web pages on the Internet.

In this way, a web crawler is a crawler, a program that crawls web pages.

The basic operation of a web crawler is to crawl web pages.

So how can you get the page you want as you wish?

Let’s start with the URL.

2. The process of browsing the webpage

The process of crawling the webpage is actually the same as how readers usually use IE browser to browse the webpage.

For example, you enter the address www.baidu.com in the address bar of the browser.

The process of opening a web page is actually that the browser, as a browsing "client", sends a request to the server, "grabs" the server-side files locally, and then interprets and displays them.

HTML is a markup language that uses tags to mark content and parse and differentiate it.

The function of the browser is to parse the obtained HTML code, and then convert the original code into the website page we see directly.

3. Concepts and examples of URI and URL

To put it simply, URL is on the browser side Enter the string http://www.baidu.com.

Before understanding URL, we must first understand the concept of URI.

What is a URI?

Every resource available on the Web, such as HTML documents, images, video clips, programs, etc., is located by a Universal Resource Identifier (URI).

URI usually consists of three parts:

①The naming mechanism for accessing resources;

②The host name where the resource is stored;

③The name of the resource itself , represented by a path.

For example, the following URI:
http://www.why.com.cn/myhtml/html1223/

We can interpret it like this:

①This is a resource that can be accessed through the HTTP protocol,

②It is located on the host www.webmonkey.com.cn,

③Accessed through the path "/html/html40".

4. Understanding and examples of URL

URL is a subset of URI. It is the abbreviation of Uniform Resource Locator, translated as "Uniform Resource Locator".

In layman's terms, URL is a string describing information resources on the Internet. It is mainly used in various WWW client programs and server programs.

URLs can be used to describe various information resources in a unified format, including files, server addresses and directories, etc.

The general format of URL is (with square brackets [] optional):

protocol :// hostname[:port] / path / [;parameters][?query]#fragment

The URL format consists of three parts:

①The first part is the protocol (or service method) ).

②The second part is the IP address of the host where the resource is stored (sometimes including the port number).

③The third part is the specific address of the host resource, such as directory and file name, etc.

The first part and the second part are separated by the "://" symbol,

The second part and the third part are separated by the "/" symbol.

The first and second parts are indispensable, and the third part can sometimes be omitted.

5. Simple comparison of URL and URI

URI is a lower-level abstraction of URL, a kind of string Text standards.

In other words, URI belongs to the parent class, and URL belongs to the subclass of URI. URL is a subset of URI. The definition of

URI is: Uniform Resource Identifier; the definition of

URL is: Uniform Resource Locator.

The difference between the two is that URI represents the path to the request server and defines such a resource.

The URL also describes how to access this resource (http://).

Let’s take a look at a small example of two URLs.

1. URL example of HTTP protocol:
Use Hypertext Transfer Protocol HTTP to provide resources for hypertext information services.

Example: http://www.peopledaily.com.cn/channel/welcome.htm

The computer domain name is www.peopledaily.com.cn.

The hypertext file (file type is .html) is welcome.htm in the directory /channel.

This is a computer from the People's Daily of China.

Example: http://www.rol.cn.NET/talk/talk1.htm

The computer domain name is www.rol.cn.Net.

The hypertext file (file type is .html) is talk1.htm in the directory/talk.

This is the address of Ruide Chat Room. You can enter the first room of Ruide Chat Room from here.

2. File URL
When using URL to represent a file, the server mode is represented by file, followed by the host IP address, file access path (i.e. directory), file name and other information.

Sometimes directory and file names can be omitted, but the "/" symbol cannot be omitted.

Example: file://ftp.yoyodyne.com/pub/files/foobar.txt

The above URL represents the pub/files/ directory stored on the host ftp.yoyodyne.com A file under, the file name is foobar.txt.

Example: file://ftp.yoyodyne.com/pub

represents the directory/pub on the host ftp.yoyodyne.com.

Example: file://ftp.yoyodyne.com/

represents the root directory of the host ftp.yoyodyne.com.

The main processing object of the crawler is the URL. It obtains the required file content based on the URL address, and then further processes it.

Therefore, accurately understanding URLs is crucial to understanding web crawlers.

The above is [Python] web crawler (1): crawling the meaning of web pages and the basic composition of URLs. For more related content, please pay attention to the PHP Chinese website (www.php.cn)!