How to Deploy Apache Airflow on Vultr Using Anaconda

This article guides you through deploying a secure Airflow application within a Conda environment, leveraging Nginx as a reverse proxy and Let's Encrypt for SSL certificates. Airflow, a powerful workflow management tool, is deployed on a Vultr server for scalability and reliability.

Airflow simplifies the definition, scheduling, and monitoring of complex workflows using Directed Acyclic Graphs (DAGs). Its open-source nature ensures community support and continuous improvement.

(This is a sponsored article by Vultr, a leading cloud computing platform offering scalable solutions worldwide.)

Deploying Your Server on Vultr:

1. Access the Vultr Customer Portal and create an account.

2. Navigate to "Products" and select "Compute."

3. Choose "Deploy Server," selecting "Cloud Compute" as the server type.

4. Specify your preferred location and choose "Anaconda" from the marketplace applications.

   

5. Select a plan and any additional features.

6. Click "Deploy Now."

   

Setting Up a Vultr Managed PostgreSQL Database:

Next, create a Vultr-managed PostgreSQL database and two new databases within it ("airflow-pgsql" and "airflow-celery") for Airflow's metadata and Celery results, respectively. The steps involve navigating to "Databases," adding a PostgreSQL database, and then adding the two new databases under "Users & Databases." Screenshots illustrate each step. (Screenshots similar to the originals would be included here, referencing the original image URLs.)

Conda Environment Setup and Airflow Installation:

1. Verify your Conda version (conda --ver).

2. Create a Conda environment (conda create -n airflow python=3.8).

3. Activate the environment (conda activate airflow).

4. Install Redis (apt install -y redis-server), enable it (sudo systemctl enable redis-server), and check its status (sudo systemctl status redis-server).

   

5. Install pip (conda install pip).

6. Install necessary dependencies (pip install psycopg2-binary virtualenv redis).

7. Install Airflow (pip install "apache-airflow[celery]==2.8.1" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.8.txt").

Connecting Airflow to the Vultr Database:

1. Set the AIRFLOW__DATABASE__SQL_ALCHEMY_CONN environment variable using your database credentials (referencing the airflow-pgsql database). (Screenshot similar to the original would be included here, referencing the original image URL.)
2. Initialize the Airflow metadata database (airflow db init).
3. Configure airflow.cfg, setting the executor to CeleryExecutor, updating sql_alchemy_conn (using airflow-pgsql), worker and trigger log ports (8794 and 8795), broker_url (redis://localhost:6379/0), and result_backend (using airflow-celery). (Screenshot similar to the original would be included here, referencing the original image URL.)
4. Create an Airflow user (airflow users create ...). (Remember to replace placeholders with actual values).

Daemonizing Airflow Services:

Create and enable systemd service files for the Airflow webserver, scheduler, and Celery worker to ensure they run persistently in the background. This involves creating .service files (e.g., airflow-webserver.service) with appropriate ExecStart paths and user/group settings, then enabling and starting them using systemctl. (Screenshots similar to the originals would be included here, referencing the original image URLs.)

Nginx Reverse Proxy Configuration:

Set up Nginx as a reverse proxy to improve security and scalability. This includes installing Nginx, creating a virtual host configuration file (airflow.conf) to proxy requests to the Airflow webserver (port 8080), and enabling the configuration. Remember to configure firewall rules to allow HTTP (port 80) and HTTPS (port 443) traffic. (Screenshot similar to the original would be included here, referencing the original image URL.)

Let's Encrypt SSL Certificate:

Finally, obtain and install an SSL certificate from Let's Encrypt using Certbot. This secures your Airflow application with HTTPS. The process involves installing Certbot, obtaining the certificate (certbot --nginx -d airflow.example.com), and testing auto-renewal. (Screenshot similar to the original would be included here, referencing the original image URL.)

Conclusion:

This comprehensive guide details deploying a robust and secure Airflow application on Vultr, using Conda, Nginx, and Let's Encrypt. The result is a production-ready, scalable, and secure workflow management system.

The above is the detailed content of How to Deploy Apache Airflow on Vultr Using Anaconda. For more information, please follow other related articles on the PHP Chinese website!