What does python crawler do?

Python crawler refers to a crawler developed based on Python, and a web crawler is a program or script that automatically crawls World Wide Web information according to certain rules; users can use crawlers to crawl pictures, crawl videos, etc. The data you want to crawl can be obtained through the crawler as long as it can be accessed through the browser.

The operating environment of this article: linux5.9.8 system, Dell G3 computer, python3.6.4.

80% of the world's crawlers are developed based on Python. Learning crawler skills well can provide important data sources for subsequent big data analysis, mining, machine learning, etc. You can use crawlers to crawl pictures, crawl videos, and other data you want to crawl. As long as you can access the data through the browser, you can obtain it through the crawler.

What is a crawler?

Web crawlers (also known as web spiders, web robots, and more commonly known as web chasers in the FOAF community) are a type of web crawler that automatically crawls the World Wide Web according to certain rules. information program or script. Other less commonly used names include ants, autoindexers, emulators, or worms.

In fact, in layman's terms, it is to obtain the data you want on the web page through a program, that is, to automatically capture the data.

What is the essence of a crawler?

Simulate the browser to open the web page and obtain the part of the data we want in the web page

The process of the browser opening the web page:

When you enter the address in the browser, you find the server host through the DNS server and send a request to the server. The server parses and sends the results to the user's browser, including html, js, css and other file contents, which are parsed by the browser and finally presented. Give the results that the user sees on the browser

So the results of the browser that the user sees are composed of HTML code. Our crawler is to obtain this content by analyzing and filtering the HTML code to obtain us. Want resources.

[Recommended reading: Python video tutorial]

The above is the detailed content of What does python crawler do?. For more information, please follow other related articles on the PHP Chinese website!