Install Scrapy Tutorial

Scrapy is a web crawler framework based on python. It can crawl information from the network and is a good way to obtain data. So I wanted to install it and take a look.

Enter its official website, the installation introduction page

There are three installation methods, one is from pip, one is from source code compilation, and the other is from conda

according to Previous knowledge, pip is already a package management tool integrated in python. It is the simplest and clearest. I ignored the sentence in the official website introduction interface

Note that sometimes this may require solving compilation issues for some Scrapy dependencies depending on your operating system

As a result, many errors were reported during the compilation phase, and one after another was solved.

Then I gave up and compiled from the source code. Just like pip, there were a lot of compilation errors.

There is no other way, so I went to see conda and downloaded a miniconda, which cost more than 60MB. After careful study, the results are really cool.

Perhaps python has also noticed that its package needs to be compiled after downloading it. Compilation needs to rely on the environment configuration of its own OS, which often causes errors.

miniconda is an integrated environment that has already installed python. It is equivalent to downloading and installing miniconda, which means downloading the basic python core program. Then you can use the conda command to download the package that conda has compiled. Make function extensions. That is to say, the scrapy package and the lxml, twisted and other compiled packages it depends on have been compiled. Then just download it and use it directly.

The above is the detailed content of Install Scrapy Tutorial. For more information, please follow other related articles on the PHP Chinese website!