What libraries need to be installed to learn Python web crawler?
The following are the relevant libraries involved in Python crawlers
Request library, parsing library, storage library, tool library
1. Request library: urllib/re/requests
(1) urllib/re is the library that comes with python by default. It can be verified by the following command:
No error message is output, indicating that the environment is normal
(2) requests installation
2.1 Open CMD, enter pip3 install requests
2.2 Wait for installation , Verification
(3) selenium installation (driving the browser for website access behavior)
3.1 Open CMD and enter pip3 install selenium
3.2 Install chromedriver
Website: https://npm.taobao.org/
Extract the downloaded compressed package and put the exe into D:\Python3.6.0 \Scripts\
This path only needs to be in the PATH variable
3.3 Wait for the installation to be completed, verify
and press Enter to pop up chrome browser interface
3.4 Install other browsers
Interfaceless browser phantomjs
Download URL: http://phantomjs.org/
Download After completion, unzip it, put the entire directory into D:\Python3.6.0\Scripts\, and add the path to the bin directory to the PATH variable
Verification:
Open CMD
2. Parsing library:
2.1 lxml (XPATH)
Open CMD and enter pip3 install lxml or download from https://pypi.python.org, For example, lxml-4.1.1-cp36-cp36m-win_amd64.whl (md5), first download the whl file, and execute pip3 install on the command line. File name.whl
2.2 beautifulsoup
To open CMD, you need to install lxml first
pip3 install beautifulsoup4
2.3 pyquery (similar to jquery syntax)
Open CMD , pip3 install pyquery
Verify the installation results
3. Repository
3.1 pymysql (operating MySQL, relational database)
Installation: pip3 install pymysql, test after installation:
3.2 pymongo (operating MongoDB, key-value)
Installation pip3 install pymongo
Verification
The above is the detailed content of What packages should be installed for python crawler?. For more information, please follow other related articles on the PHP Chinese website!
How to Use Python to Find the Zipf Distribution of a Text FileMar 05, 2025 am 09:58 AMThis tutorial demonstrates how to use Python to process the statistical concept of Zipf's law and demonstrates the efficiency of Python's reading and sorting large text files when processing the law. You may be wondering what the term Zipf distribution means. To understand this term, we first need to define Zipf's law. Don't worry, I'll try to simplify the instructions. Zipf's Law Zipf's law simply means: in a large natural language corpus, the most frequently occurring words appear about twice as frequently as the second frequent words, three times as the third frequent words, four times as the fourth frequent words, and so on. Let's look at an example. If you look at the Brown corpus in American English, you will notice that the most frequent word is "th
How Do I Use Beautiful Soup to Parse HTML?Mar 10, 2025 pm 06:54 PMThis article explains how to use Beautiful Soup, a Python library, to parse HTML. It details common methods like find(), find_all(), select(), and get_text() for data extraction, handling of diverse HTML structures and errors, and alternatives (Sel
Image Filtering in PythonMar 03, 2025 am 09:44 AMDealing with noisy images is a common problem, especially with mobile phone or low-resolution camera photos. This tutorial explores image filtering techniques in Python using OpenCV to tackle this issue. Image Filtering: A Powerful Tool Image filter
How to Work With PDF Documents Using PythonMar 02, 2025 am 09:54 AMPDF files are popular for their cross-platform compatibility, with content and layout consistent across operating systems, reading devices and software. However, unlike Python processing plain text files, PDF files are binary files with more complex structures and contain elements such as fonts, colors, and images. Fortunately, it is not difficult to process PDF files with Python's external modules. This article will use the PyPDF2 module to demonstrate how to open a PDF file, print a page, and extract text. For the creation and editing of PDF files, please refer to another tutorial from me. Preparation The core lies in using external module PyPDF2. First, install it using pip: pip is P
How to Cache Using Redis in Django ApplicationsMar 02, 2025 am 10:10 AMThis tutorial demonstrates how to leverage Redis caching to boost the performance of Python applications, specifically within a Django framework. We'll cover Redis installation, Django configuration, and performance comparisons to highlight the bene
How to Perform Deep Learning with TensorFlow or PyTorch?Mar 10, 2025 pm 06:52 PMThis article compares TensorFlow and PyTorch for deep learning. It details the steps involved: data preparation, model building, training, evaluation, and deployment. Key differences between the frameworks, particularly regarding computational grap
How to Implement Your Own Data Structure in PythonMar 03, 2025 am 09:28 AMThis tutorial demonstrates creating a custom pipeline data structure in Python 3, leveraging classes and operator overloading for enhanced functionality. The pipeline's flexibility lies in its ability to apply a series of functions to a data set, ge
Introduction to Parallel and Concurrent Programming in PythonMar 03, 2025 am 10:32 AMPython, a favorite for data science and processing, offers a rich ecosystem for high-performance computing. However, parallel programming in Python presents unique challenges. This tutorial explores these challenges, focusing on the Global Interprete
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
SublimeText3 Linux new version
SublimeText3 Linux latest version
SecLists
SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.
WebStorm Mac version
Useful JavaScript development tools
SublimeText3 English version
Recommended: Win version, supports code prompts!
