[Python] Web Crawler (11): Show your sword! The crawler framework Scrapy makes its debut!

The previous ten chapters of crawler notes have successively recorded some simple Python crawler knowledge,

can be used to solve simple Tieba downloads, and the calculation of grade points is naturally a no-brainer.

But if you want to download a large amount of content in batches, such as all the questions and answers on Zhihu, it seems a bit difficult.

Ever since, the crawler framework Scrapy has appeared!

Scrapy = Scrach+Python. The word Scrach means scraping.

For the time being, we can call it: small scratch.

Xiaozhuazhao’s official website address: Click me, click me.

#Then let’s briefly demonstrate the installation process of Scrapy.

For specific procedures, please refer to: Official website tutorial

Friendly reminder: Be sure to download according to the Python version, otherwise you will be reminded that Python cannot be found during installation. It is recommended that you install 32-bit because some 64-bit versions of essential software are difficult to find.

1. Install Python (32-bit recommended)

It is recommended to install Python2.7.x, 3.x does not seem to be supported yet.

After installation, remember to configure the environment and add the python directory and the Scripts directory under the python directory to the Path of the system environment variable.

Enter python in cmd. If version information appears, the configuration is complete.

2. Install lxml

lxml is a library written in Python that can process XML quickly and flexibly. Click here to select the corresponding Python version to install.

3. Install setuptools

is used to install the egg file. Click here to download the corresponding version of setuptools for python2.7.

4. Install zope.interface

You can use the setuptools downloaded in the third step to install the egg file. There is also an exe version now. Click here to download.

5. Install Twisted

Twisted is an event-driven network engine framework implemented in Python. Click here to download.

6. Install pyOpenSSL

pyOpenSSL is Python’s OpenSSL interface, click here to download.

7. Install win32py

Provides win32api, click here to download

8 .Install Scrapy

Finally it’s time to get excited! After installing so many widgets, it’s finally the protagonist’s turn.

Directly enter easy_install scrapy in cmd and press Enter.

9. Check the installation

Open a cmd window and execute the scrapy command anywhere. If you get the following page, it means the environment configuration is successful.

The above is [Python] Web Crawler (11): Show your sword! The crawler framework Scrapy makes its debut! For more related content, please pay attention to the PHP Chinese website (www.php.cn)!