Building a Document Retrieval & Q&A System with OpenAI and Streamlit

Hello, Dev Community! ?

Today, I’m excited to walk you through my project: EzioDevIo RAG (Retrieval-Augmented Generation). This system allows users to upload PDF documents, ask questions based on their content, and receive real-time answers generated by OpenAI's GPT-3.5 Turbo model. This is particularly useful for navigating large documents or quickly extracting relevant information. ??

You can find the complete code on my GitHub: EzioDevIo RAG Project. Let’s dive into the project and break down each step!

? Dive into the full codebase and setup instructions in the EzioDevIo RAG Project GitHub Repository!

Project Overview

What You’ll Learn

1. How to integrate OpenAI’s language models with PDF document retrieval.
2. How to create a user-friendly interface using Streamlit.
3. How to containerize the application with Docker for easy deployment. Project Features

• Upload PDFs and get information from them.
• Ask questions based on the content of the uploaded PDFs.
• Real-time responses generated by OpenAI’s gpt-3.5-turbo model.
• Easy deployment with Docker for scalability.

*Here’s the final structure of our project directory: *

RAG-project/
├── .env                       # Environment variables (API key)
├── app.py                     # Streamlit app for the RAG system
├── document_loader.py         # Code for loading and processing PDF documents
├── retriever.py               # Code for indexing and retrieving documents
├── main.py                    # Main script for RAG pipeline
├── requirements.txt           # List of required libraries
├── Dockerfile                 # Dockerfile for containerizing the app
├── .gitignore                 # Ignore sensitive and unnecessary files
├── data/
│   └── uploaded_pdfs/         # Folder to store uploaded PDFs
└── images/
    └── openai_api_setup.png   # Example image for OpenAI API setup instructions

Step 1: Setting Up the Project

Prerequisites

Make sure you have the following:

• Python 3.8 : To run the application locally.
• OpenAI API Key: You’ll need this to access OpenAI’s models. Sign up at OpenAI API to get your key.
• Docker: Optional, but recommended for containerizing the application for deployment.

Step 2: Clone the Repository and Set Up the Virtual Environment

2.1. Clone the Repository
Start by cloning the project repository from GitHub and navigating into the project directory.

git clone https://github.com/EzioDEVio/RAG-project.git
cd RAG-project

2.2. Set Up a Virtual Environment
To isolate project dependencies, create and activate a virtual environment. This helps prevent conflicts with other projects’ packages.

python -m venv venv
source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

2.3. Install Dependencies
Install the required Python libraries listed in requirements.txt. This includes OpenAI for the language model, Streamlit for the UI, PyMuPDF for PDF handling, and FAISS for efficient similarity search.

pip install -r requirements.txt

2.4. Configure Your OpenAI API Key
Create a .env file in the project root directory. This file will store your OpenAI API key securely. Add the following line to the file, replacing your_openai_api_key_here with your actual API key:

RAG-project/
├── .env                       # Environment variables (API key)
├── app.py                     # Streamlit app for the RAG system
├── document_loader.py         # Code for loading and processing PDF documents
├── retriever.py               # Code for indexing and retrieving documents
├── main.py                    # Main script for RAG pipeline
├── requirements.txt           # List of required libraries
├── Dockerfile                 # Dockerfile for containerizing the app
├── .gitignore                 # Ignore sensitive and unnecessary files
├── data/
│   └── uploaded_pdfs/         # Folder to store uploaded PDFs
└── images/
    └── openai_api_setup.png   # Example image for OpenAI API setup instructions

? Tip: Make sure .env is added to your .gitignore file to avoid exposing your API key if you push your project to a public repository.

Step 3: Understanding the Project Structure
Here’s a quick overview of the directory structure to help you navigate the code:
Here’s a quick overview of the directory structure to help you navigate the code:

git clone https://github.com/EzioDEVio/RAG-project.git
cd RAG-project

Each file has a specific role:

• app.py: Manages the Streamlit interface, allowing users to upload files and ask questions.
• document_loader.py: Handles loading and processing of PDFs using PyMuPDF.
• retriever.py: Uses FAISS to index document text and retrieve relevant sections based on user queries.
• main.py: Ties everything together, including calling OpenAI’s API to generate responses.

Step 4: Building the Core Code
Now, let’s dive into the main components of the project.

4.1. Loading Documents (document_loader.py)
The document_loader.py file is responsible for extracting text from PDFs. Here, we use the PyMuPDF library to process each page in the PDF and store the text.

python -m venv venv
source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

Explanation: This function reads all PDF files in a specified folder, extracts the text from each page, and adds the text to a list of dictionaries. Each dictionary represents a document with its text and filename.

4.2. Document Indexing and Retrieval (retriever.py)
FAISS (Facebook AI Similarity Search) helps us to perform similarity searches. We use it to create an index of the document embeddings, which allows us to retrieve relevant sections when users ask questions.

pip install -r requirements.txt

Explanation:

create_index: Converts document text into embeddings using OpenAIEmbeddings and creates an index with FAISS.
retrieve_documents: Searches for relevant document sections based on the user query.

4.3. Generating Responses (main.py)
This module processes user queries, retrieves relevant documents, and generates answers using OpenAI’s language model.

OPENAI_API_KEY=your_openai_api_key_here

Explanation:

generate_response: Creates a prompt with context from retrieved documents and the user’s query, then sends it to OpenAI’s API. The response is then returned as the answer.

Step 5: Creating the Streamlit Interface (app.py)
Streamlit provides an interactive front end, making it easy for users to upload files and ask questions.

RAG-project/
├── .env                       # Environment variables (API key)
├── app.py                     # Streamlit app for the RAG system
├── document_loader.py         # Code for loading and processing PDF documents
├── retriever.py               # Code for indexing and retrieving documents
├── main.py                    # Main script for RAG pipeline
├── requirements.txt           # List of required libraries
├── Dockerfile                 # Dockerfile for containerizing the app
├── .gitignore                 # Ignore sensitive and unnecessary files
├── data/
│   └── uploaded_pdfs/         # Folder to store uploaded PDFs
└── images/
    └── openai_api_setup.png   # Example image for OpenAI API setup instructions

Explanation:

• This code creates a simple UI with Streamlit, allowing users to upload PDFs and type questions.
• When users click "Get Answer," the app retrieves relevant documents and generates an answer.

Step 6: Dockerizing the Application
Docker allows you to package the app into a container, making it easy to deploy.

Dockerfile

RAG-project/
├── .env                       # Environment variables (API key)
├── app.py                     # Streamlit app for the RAG system
├── document_loader.py         # Code for loading and processing PDF documents
├── retriever.py               # Code for indexing and retrieving documents
├── main.py                    # Main script for RAG pipeline
├── requirements.txt           # List of required libraries
├── Dockerfile                 # Dockerfile for containerizing the app
├── .gitignore                 # Ignore sensitive and unnecessary files
├── data/
│   └── uploaded_pdfs/         # Folder to store uploaded PDFs
└── images/
    └── openai_api_setup.png   # Example image for OpenAI API setup instructions

Explanation:

We use a multi-stage build to keep the final image lean.
The application runs as a non-root user for security.

Running the Docker Container

1. Build the Docker Image:

git clone https://github.com/EzioDEVio/RAG-project.git
cd RAG-project

1. Run the Container:

python -m venv venv
source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

Step 7: Setting Up CI/CD with GitHub Actions
For production readiness, add a CI/CD pipeline to build, test, and scan Docker images. You can find the .github/workflows file in the repository for this setup.

Final Thoughts
This project combines OpenAI’s language model capabilities with document retrieval to create a functional and interactive tool. If you enjoyed this project, please star the GitHub repository and follow me here on Dev Community. Let’s build more amazing projects together! ?

? View the GitHub Repository ? EzioDevIo RAG Project GitHub Repository!

The above is the detailed content of Building a Document Retrieval & Q&A System with OpenAI and Streamlit. For more information, please follow other related articles on the PHP Chinese website!