How does Scrapy automate deployment to the server?

Scrapy is one of the most popular crawler frameworks in Python. It can help users quickly build crawlers and improve the efficiency of crawlers. The automatic deployment of Scrapy to the server can make it more convenient for users to deploy and manage crawler programs. This article will introduce how to automatically deploy Scrapy to the server.

1. Preparation tools

To automatically deploy Scrapy to the server, we need some tools, these tools include:

1. Git: version control tool, used for Manage code between local and server.
2. SSH: Secure Shell, securely transfer data and execute commands between local and remote servers.
3. Virtualenv: Python virtual environment management tool, which can make our Scrapy deployment environment clearer and independent.
4. Supervisor: Process control system, used to manage and monitor the Scrapy crawler process on the server.

These tools are very common. If they are not installed yet, you can install them according to the official documentation.

2. Create a project

Before automated Scrapy deployment, we first need to create a Scrapy project locally. You can use the Scrapy command line tool to create:

scrapy startproject myproject

This will create a Scrapy project named myproject, which includes some default code and directory structure. Next, we can write Spiders, Pipelines and other components.

In order to make our Scrapy project more convenient to deploy and run on the server, we can create a requirements.txt file in the project root directory to manage the project dependencies Python libraries and versions. This file can be created through pip, for example:

pip freeze > requirements.txt

This will automatically generate a requirements.txt file, which contains all Python libraries installed in the current system and their version information . We need to manually remove the unnecessary libraries in it and keep the necessary libraries like Scrapy and other related libraries and tools. After determining the dependencies, we can use pip to install these dependencies in the virtual environment:

pip install -r requirements.txt

3. Configure the server

Before starting the deployment, we need to install the required components on the server. Taking Ubuntu as an example, we need to install Git, SSH, Virtualenv and Supervisor. It can be installed through the following command:

sudo apt-get update
sudo apt-get install git ssh virtualenv supervisor

After the installation is completed, we need to create a new user on the server. This user will play an important role in subsequent deployment and operation and will have access to the Scrapy crawler. You can use the following command to create a new user named myuser:

sudo adduser myuser

Next, we need to create a new directory to save Scrapy deployment files and related configurations. On the server, you can use the following command to create a directory /srv/myproject:

sudo mkdir /srv/myproject
sudo chown myuser:myuser /srv/myproject

4. Set up the Git warehouse and SSH

Next, we need to Scrapy projects are uploaded to a Git repository and deployed using SSH. Locally, we can use the following command to upload all the code in the Scrapy project to the Git repository:

git init
git add .
git commit -m "Initial commit"
git remote add origin ssh://myuser@myserver.com/srv/myproject.git
git push -u origin master

This code uploads the Scrapy project to the remote server and saves it in /srv/myproject.git Under contents.

Next, we need to configure SSH on the server so that we can use SSH to connect to the Git repository and perform related operations. We can create an SSH key for this and add the public key to the authorized_keys file on the server side.

First, we can create a new SSH key locally using the following command:

ssh-keygen

This will create a pair of public and private keys. Next, we need to add the public key to the authorized_keys file on the server side:

ssh myuser@myserver.com "mkdir -p ~/.ssh && chmod 0700 ~/.ssh && echo 'PUBLIC_KEY' >> ~/.ssh/authorized_keys && chmod 0600 ~/.ssh/authorized_keys"

Please replace PUBLIC_KEY with the public key on your local computer.

Now we can use SSH to connect to the server and perform operations.

5. Deploy the Scrapy project

Now, we are ready to automatically deploy the Scrapy project on the server. To do this, we need to create a new virtual environment on the server and install Scrapy and other required dependent libraries:

mkdir /srv/myproject/env
virtualenv /srv/myproject/env
source /srv/myproject/env/bin/activate
pip install scrapy supervisor

Create a Scrapy project working directory on the server, clone the Scrapy project from the Git repository, and Create a supervisord.conf file for configuring process management:

mkdir /srv/myproject/src
cd /srv/myproject/src
git clone ssh://myuser@myserver.com/srv/myproject.git .
cp /srv/myproject/env/bin/supervisord /srv/myproject/env/bin/supervisord.conf /etc
sudo supervisorctl reread
sudo supervisorctl update

This will clone the Scrapy project to the server and place it in the /srv/myproject directory Create the supervisord.conf file. We can edit the supervisord.conf file to start the Scrapy crawler:

[program:myproject]
command=/srv/myproject/env/bin/scrapy crawl myspider
directory=/srv/myproject/src
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true

where the command parameter is used to start the Scrapy crawler, and the directory parameter is used to Specify the working directory. The autostart and autorestart parameters are used to automatically restart the Scrapy crawler after it stops. The stopasgroup and killasgroup parameters are used to stop the process. Stop all related processes at the same time.

Finally, we can use the following command to start the Scrapy crawler:

sudo supervisorctl start myproject

In this way, the Scrapy crawler can be deployed to the server and run automatically.

Summarize

Automated deployment of Scrapy to the server is a very convenient and efficient method. We can upload the code to the remote Git repository and connect to the server through SSH for deployment and management. By using Virtualenv and Supervisor, we can better control the project's environment and process, and let Scrapy crawlers run automatically on the server.

The above is the detailed content of How does Scrapy automate deployment to the server?. For more information, please follow other related articles on the PHP Chinese website!