80% of the world's crawlers are developed based on Python. Learning crawler skills can provide important data sources for subsequent big data analysis, mining, machine learning, etc.
Python crawler needs to install related libraries:
Libraries involved in python crawler:
Request library, parsing library, storage library, tool library
1. Request library: urllib/re/requests
(1) urllib/re is the library that comes with python by default and can be verified by the following command:
No error message is output, indicating that the environment is normal
(2) requests installation
2.1 Open CMD and enter
pip3 install requests
2.2 Wait for installation and verify
(3) Selenium installation (drives the browser for website access behavior)
3.1 Open CMD and enter
pip3 install selenium
3.2 Install chromedriver
Website: https://npm.taobao.org/
Decompress the downloaded compressed package and put the exe into D:\Python3.6.0\Scripts\
This path only needs to be in the PATH variable
3.3 After the installation is completed, verify
Press Enter and the chrome browser interface will pop up
3.4 Install other browsers
Interfaceless browser phantomjs
Download URL: http://phantomjs.org/
After downloading, unzip and put the entire directory Go to D:\Python3.6.0\Scripts\ and add the path to the bin directory to the PATH variable
Verification:
Open CMD
phantomjs console.log('phantomjs') CTRL+C python from selenium import webdriver driver = webdriver.PhantomJS() dirver.get('http://www.baidu.com') driver.page_source
2. Parsing library:
2.1 lxml (XPATH)
Open CMD
pip3 install lxml
or download from https://pypi.python.org, for example, lxml-4.1.1-cp36- cp36m-win_amd64.whl (md5), first download the whl file
pip3 install 文件名.whl
2.2 beautifulsoup
Open CMD, you need to install lxml
pip3 install beautifulsoup4
Verification
python from bs4 import BeautifulSoup soup = BeautifulSoup('<html></html>','lxml')
2.3 pyquery (similar to jquery syntax)
Open CMD
pip3 install pyquery
Verify the installation results
python from pyquery import PyQuery as pq doc = pq('<html>hi</html>') result = doc('html').text() result
3. Repository
3.1 pymysql (operating MySQL, relational database)
Installation:
pip3 install pymysql
Test after installation:
3.2 pymongo (operating MongoDB, key-value)
Installation
pip3 install pymongo
Verification
python
import pymongo
client = pymongo.MongoClient('localhost')
db = client['testdb']
db['table'].insert({'name':'bob'})
db['table'].find_one({'name':'bob'}) 
3.3 redis (distributed crawler, maintain crawling queue)
Installation:
pip3 install redis
Verification:
4. Tool library
4.1 flask (WEB library)
pip3 install flask
4.2 Django (distributed crawler maintenance system)
pip3 install django
4.3 jupyter (notepad running on the web page, supports markdown, and can run code on the web page)
pip3 install jupyter
Verification:
After opening CMD
jupyter notebook
, you can directly run the code on the web page Create notepad, code blocks and Markdown blocks, support printing
[Related recommendations]
1. Python crawler library and related tools
2. Python crawler introductory tutorial
The above is the detailed content of What needs to be installed for python crawler. For more information, please follow other related articles on the PHP Chinese website!
Why are arrays generally more memory-efficient than lists for storing numerical data?May 05, 2025 am 12:15 AMArraysaregenerallymorememory-efficientthanlistsforstoringnumericaldataduetotheirfixed-sizenatureanddirectmemoryaccess.1)Arraysstoreelementsinacontiguousblock,reducingoverheadfrompointersormetadata.2)Lists,oftenimplementedasdynamicarraysorlinkedstruct
How can you convert a Python list to a Python array?May 05, 2025 am 12:10 AMToconvertaPythonlisttoanarray,usethearraymodule:1)Importthearraymodule,2)Createalist,3)Usearray(typecode,list)toconvertit,specifyingthetypecodelike'i'forintegers.Thisconversionoptimizesmemoryusageforhomogeneousdata,enhancingperformanceinnumericalcomp
Can you store different data types in the same Python list? Give an example.May 05, 2025 am 12:10 AMPython lists can store different types of data. The example list contains integers, strings, floating point numbers, booleans, nested lists, and dictionaries. List flexibility is valuable in data processing and prototyping, but it needs to be used with caution to ensure the readability and maintainability of the code.
What is the difference between arrays and lists in Python?May 05, 2025 am 12:06 AMPythondoesnothavebuilt-inarrays;usethearraymoduleformemory-efficienthomogeneousdatastorage,whilelistsareversatileformixeddatatypes.Arraysareefficientforlargedatasetsofthesametype,whereaslistsofferflexibilityandareeasiertouseformixedorsmallerdatasets.
What module is commonly used to create arrays in Python?May 05, 2025 am 12:02 AMThemostcommonlyusedmoduleforcreatingarraysinPythonisnumpy.1)Numpyprovidesefficienttoolsforarrayoperations,idealfornumericaldata.2)Arrayscanbecreatedusingnp.array()for1Dand2Dstructures.3)Numpyexcelsinelement-wiseoperationsandcomplexcalculationslikemea
How do you append elements to a Python list?May 04, 2025 am 12:17 AMToappendelementstoaPythonlist,usetheappend()methodforsingleelements,extend()formultipleelements,andinsert()forspecificpositions.1)Useappend()foraddingoneelementattheend.2)Useextend()toaddmultipleelementsefficiently.3)Useinsert()toaddanelementataspeci
How do you create a Python list? Give an example.May 04, 2025 am 12:16 AMTocreateaPythonlist,usesquarebrackets[]andseparateitemswithcommas.1)Listsaredynamicandcanholdmixeddatatypes.2)Useappend(),remove(),andslicingformanipulation.3)Listcomprehensionsareefficientforcreatinglists.4)Becautiouswithlistreferences;usecopy()orsl
Discuss real-world use cases where efficient storage and processing of numerical data are critical.May 04, 2025 am 12:11 AMIn the fields of finance, scientific research, medical care and AI, it is crucial to efficiently store and process numerical data. 1) In finance, using memory mapped files and NumPy libraries can significantly improve data processing speed. 2) In the field of scientific research, HDF5 files are optimized for data storage and retrieval. 3) In medical care, database optimization technologies such as indexing and partitioning improve data query performance. 4) In AI, data sharding and distributed training accelerate model training. System performance and scalability can be significantly improved by choosing the right tools and technologies and weighing trade-offs between storage and processing speeds.
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Zend Studio 13.0.1
Powerful PHP integrated development environment
DVWA
Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software
SublimeText3 English version
Recommended: Win version, supports code prompts!
WebStorm Mac version
Useful JavaScript development tools
SublimeText3 Chinese version
Chinese version, very easy to use
