Building a Weather Data Analytics Pipeline with AWS and OpenWeatherMap API

This blog post guides you through building a weather data analytics pipeline using the OpenWeatherMap API and AWS services. The pipeline fetches weather data, stores it in S3, catalogs it with AWS Glue, and allows querying with Amazon Athena.

Project Overview

This project creates a scalable data pipeline for fetching weather data from multiple cities, storing it in AWS S3, cataloging it via AWS Glue, and enabling querying using Amazon Athena.

Initial Architecture & Architecture Diagrams

Project Structure & Prerequisites

Before starting, ensure you have:

1. Docker: Installed locally.
2. AWS Account: With necessary permissions (S3 buckets, Glue databases, Glue crawlers).
3. OpenWeatherMap API Key: Obtained from OpenWeatherMap.

Setup Guide

1. Clone the Repository:

   git clone https://github.com/Rene-Mayhrem/weather-insights.git
   cd weather-data-analytics

2. Create a .env File: Create a .env file in the root directory with your AWS credentials and API key:

   <code>AWS_ACCESS_KEY_ID=<your-access-key-id>
   AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
   AWS_REGION=us-east-1
   S3_BUCKET_NAME=<your-s3-bucket-name>
   OPENWEATHER_API_KEY=<your-openweather-api-key></your-openweather-api-key></your-s3-bucket-name></your-secret-access-key></your-access-key-id></code>

3. Create cities.json: Create cities.json listing the cities:

   {
     "cities": [
       "London",
       "New York",
       "Tokyo",
       "Paris",
       "Berlin"
     ]
   }

4. Docker Compose: Build and run:

   docker compose run terraform init
   docker compose run python

Usage

1. Verify Infrastructure: Check if Terraform created the AWS resources (S3, Glue database, Glue crawler) in the AWS console.

2. Verify Data Upload: Confirm the Python script uploaded weather data (JSON files) to your S3 bucket via the AWS console.

3. Run Glue Crawler: The Glue crawler should run automatically; verify its execution and data cataloging in the Glue console.

4. Query with Athena: Use the AWS Management Console to access Athena and run SQL queries on the cataloged data.

Key Components

• Docker: Provides consistent environments for Python and Terraform.
• Terraform: Manages AWS infrastructure (S3, Glue, Athena).
• Python: Fetches and uploads weather data to S3.
• Glue: Catalogs S3 data.
• Athena: Queries the cataloged data.

Conclusion

This guide helps you build a scalable weather data analytics pipeline using AWS and OpenWeatherMap. The pipeline can be easily extended to include more cities or data sources.

The above is the detailed content of Building a Weather Data Analytics Pipeline with AWS and OpenWeatherMap API. For more information, please follow other related articles on the PHP Chinese website!