What libraries should be installed for python crawler?-Python Tutorial-php.cn

Home

Backend Development

Python Tutorial

What libraries should be installed for python crawler?

silencement

Jun 24, 2019 pm 05:35 PM

python crawler

What libraries need to be installed to learn Python web crawler?

What libraries should be installed for python crawler?

The following are the relevant libraries involved in Python crawlers

Request library, parsing library, storage library, tool library

1. Request library: urllib/re/requests

(1) urllib/re is the library that comes with Python by default. It can be verified by the following command:

What libraries should be installed for python crawler?

No error message is output, indicating that the environment is normal

(2) requests installation

2.1 Open CMD and enter pip3 install requests

2.2 After waiting for the installation, verify

What libraries should be installed for python crawler?

(3) selenium installation (driving the browser for website access behavior)

3.1 Open CMD and enter pip3 install selenium

3.2 Install chromedriver

Website: https://npm.taobao.org/

Unzip the downloaded compressed package and put the exe into D:\ Python3.6.0\Scripts\

This path only needs to be in the PATH variable

3.3 After the installation is completed, verify

What libraries should be installed for python crawler?

Return The chrome browser interface pops up after the car

3.4 Install other browsers

Interfaceless browser phantomjs

Download URL: http://phantomjs.org/

After downloading, unzip it, put the entire directory into D:\Python3.6.0\Scripts\, and add the path to the bin directory to the PATH variable

Verification:

Open CMD

What libraries should be installed for python crawler?

2. Parsing library:

2.1 lxml (XPATH)

Open CMD and enter pip3 install lxml or download it from https://pypi.python. org download, for example, lxml-4.1.1-cp36-cp36m-win_amd64.whl (md5), first download the whl file, and execute pip3 install on the command line. File name.whl

##2.2 beautifulsoup

Open CMD, you need to install lxml first

pip3 install beautifulsoup4

What libraries should be installed for python crawler? ##2.3 pyquery (similar to jquery syntax)

Open CMD, pip3 install pyquery

Verify the installation results

What libraries should be installed for python crawler? 3. Repository

3.1 pymysql (operating MySQL, relational database )

Installation: pip3 install pymysql, test after installation:

##3.2 pymongo (operate MongoDB, key-value) What libraries should be installed for python crawler?

Install pip3 install pymongo

Verification

3.3 redis (distributed crawler, maintain crawling queue) Installation: pip3 install redis What libraries should be installed for python crawler?

Verification:

4.1Django (distributed crawler maintenance system) pip3 install django What libraries should be installed for python crawler?

4.2jupyter (notepad running on the web page , supports markdown, and can run code on the web page) Install pip3 install jupyter

Verification: Open CMD, jupyter notebook

and then you can create notepads, code blocks and Markdown block, supports printing

The above is the detailed content of What libraries should be installed for python crawler?. For more information, please follow other related articles on the PHP Chinese website!

Statement

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

How to Use Python to Find the Zipf Distribution of a Text FileMar 05, 2025 am 09:58 AM

This tutorial demonstrates how to use Python to process the statistical concept of Zipf's law and demonstrates the efficiency of Python's reading and sorting large text files when processing the law. You may be wondering what the term Zipf distribution means. To understand this term, we first need to define Zipf's law. Don't worry, I'll try to simplify the instructions. Zipf's Law Zipf's law simply means: in a large natural language corpus, the most frequently occurring words appear about twice as frequently as the second frequent words, three times as the third frequent words, four times as the fourth frequent words, and so on. Let's look at an example. If you look at the Brown corpus in American English, you will notice that the most frequent word is "th

How Do I Use Beautiful Soup to Parse HTML?Mar 10, 2025 pm 06:54 PM

This article explains how to use Beautiful Soup, a Python library, to parse HTML. It details common methods like find(), find_all(), select(), and get_text() for data extraction, handling of diverse HTML structures and errors, and alternatives (Sel

How to Perform Deep Learning with TensorFlow or PyTorch?Mar 10, 2025 pm 06:52 PM

This article compares TensorFlow and PyTorch for deep learning. It details the steps involved: data preparation, model building, training, evaluation, and deployment. Key differences between the frameworks, particularly regarding computational grap

Mathematical Modules in Python: StatisticsMar 09, 2025 am 11:40 AM

Python's statistics module provides powerful data statistical analysis capabilities to help us quickly understand the overall characteristics of data, such as biostatistics and business analysis. Instead of looking at data points one by one, just look at statistics such as mean or variance to discover trends and features in the original data that may be ignored, and compare large datasets more easily and effectively. This tutorial will explain how to calculate the mean and measure the degree of dispersion of the dataset. Unless otherwise stated, all functions in this module support the calculation of the mean() function instead of simply summing the average. Floating point numbers can also be used. import random import statistics from fracti

Serialization and Deserialization of Python Objects: Part 1Mar 08, 2025 am 09:39 AM

Serialization and deserialization of Python objects are key aspects of any non-trivial program. If you save something to a Python file, you do object serialization and deserialization if you read the configuration file, or if you respond to an HTTP request. In a sense, serialization and deserialization are the most boring things in the world. Who cares about all these formats and protocols? You want to persist or stream some Python objects and retrieve them in full at a later time. This is a great way to see the world on a conceptual level. However, on a practical level, the serialization scheme, format or protocol you choose may determine the speed, security, freedom of maintenance status, and other aspects of the program

What are some popular Python libraries and their uses?Mar 21, 2025 pm 06:46 PM

The article discusses popular Python libraries like NumPy, Pandas, Matplotlib, Scikit-learn, TensorFlow, Django, Flask, and Requests, detailing their uses in scientific computing, data analysis, visualization, machine learning, web development, and H

Scraping Webpages in Python With Beautiful Soup: Search and DOM ModificationMar 08, 2025 am 10:36 AM

This tutorial builds upon the previous introduction to Beautiful Soup, focusing on DOM manipulation beyond simple tree navigation. We'll explore efficient search methods and techniques for modifying HTML structure. One common DOM search method is ex

How to Create Command-Line Interfaces (CLIs) with Python?Mar 10, 2025 pm 06:48 PM

This article guides Python developers on building command-line interfaces (CLIs). It details using libraries like typer, click, and argparse, emphasizing input/output handling, and promoting user-friendly design patterns for improved CLI usability.

See all articles