What libraries need to be installed for python crawler

Web crawlers (also known as web spiders, web robots, and more commonly known as web chasers in the FOAF community) are programs or scripts that automatically capture World Wide Web information according to certain rules.

#1. The environment that needs to be installed is mainly the Python environment and the database environment.

mongodb

redis

mysql

2. Some commonly used libraries

urillb

re

requests

selenium

phantomjs

lxml

beautifulsoup4

pyquery

pymysql

pymongo

redis

flask

django

jupyter

The above is the detailed content of What libraries need to be installed for python crawler. For more information, please follow other related articles on the PHP Chinese website!