使用Python和Openai構建您的第一個抹布系統

>該教程通過使用Python和Openai構建檢索增強發電（RAG）系統，為您引導您。 RAG通過從您的文檔中檢索相關信息來增強AI的響應，然後再產生答案 - 本質上，讓AI“研究”事先進行。

>

您將要學到的內容：

>從頭開始構建抹布系統。
>抹布的文檔準備和處理。
• >
>使用OpenAi嵌入。
創建一個基本的檢索系統。
• >
與OpenAI API集成。
• >

項目結構：

<code>rag-project/
│
├── src/
│   ├── __init__.py
│   ├── document_loader.py
│   ├── text_processor.py
│   ├── embeddings_manager.py
│   ├── retrieval_system.py
│   └── rag_system.py
│
├── data/
│   └── documents/
│
├── requirements.txt
├── test.py
├── README.md
└── .env</code>

步驟1：環境設置：>

創建一個虛擬環境：

（在Windows：
1. 上）python -m venv venv> venvScriptsactivate激活它：
2. source venv/bin/activate>安裝軟件包：
3. pip install openai python-dotenv numpy pandas創建
：
4. requirements.txt

<code>openai==1.12.0
python-dotenv==1.0.0
numpy==1.24.3
pandas==2.1.0</code>

configure

：
5. .env

>

<code>OPENAI_API_KEY=your_api_key_here</code>

步驟2：document loading（

）：> src/document_loader.py >

步驟3：文本處理（

import os
from typing import List

class DocumentLoader:
    def __init__(self, documents_path: str):
        self.documents_path = documents_path

    def load_documents(self) -> List[str]:
        documents = []
        for filename in os.listdir(self.documents_path):
            if filename.endswith('.txt'):
                with open(os.path.join(self.documents_path, filename), 'r') as file:
                    documents.append(file.read())
        return documents

）：

> src/text_processor.py步驟4：嵌入式創建（

）：

from typing import List

class TextProcessor:
    def __init__(self, chunk_size: int = 1000):
        self.chunk_size = chunk_size

    def split_into_chunks(self, text: str) -> List[str]:
        words = text.split()
        chunks = []
        current_chunk = []
        current_size = 0

        for word in words:
            if current_size + len(word) > self.chunk_size:
                chunks.append(' '.join(current_chunk))
                current_chunk = [word]
                current_size = len(word)
            else:
                current_chunk.append(word)
                current_size += len(word) + 1

        if current_chunk:
            chunks.append(' '.join(current_chunk))

        return chunks

src/embeddings_manager.py步驟5：檢索系統（

）：

from typing import List
import openai
import numpy as np

class EmbeddingsManager:
    def __init__(self, api_key: str):
        openai.api_key = api_key

    def create_embeddings(self, texts: List[str]) -> List[np.ndarray]:
        embeddings = []
        for text in texts:
            response = openai.embeddings.create(
                model="text-embedding-ada-002",
                input=text
            )
            embeddings.append(np.array(response.data[0].embedding))
        return embeddings

> src/retrieval_system.py>步驟6：OpenAI Integration（

）：

import numpy as np
from typing import List, Tuple

class RetrievalSystem:
    def __init__(self, chunks: List[str], embeddings: List[np.ndarray]):
        self.chunks = chunks
        self.embeddings = embeddings

    def find_similar_chunks(self, query_embedding: np.ndarray, top_k: int = 3) -> List[Tuple[str, float]]:
        similarities = []
        for i, embedding in enumerate(self.embeddings):
            similarity = np.dot(query_embedding, embedding) / (
                np.linalg.norm(query_embedding) * np.linalg.norm(embedding)
            )
            similarities.append((self.chunks[i], similarity))

        return sorted(similarities, key=lambda x: x[1], reverse=True)[:top_k]

>

>src/rag_system.py步驟7：系統用法（）：

import os
from dotenv import load_dotenv
from typing import List
import openai

from .document_loader import DocumentLoader
from .text_processor import TextProcessor
from .embeddings_manager import EmbeddingsManager
from .retrieval_system import RetrievalSystem

class RAGSystem:
    def __init__(self):
        load_dotenv()
        self.api_key = os.getenv('OPENAI_API_KEY')
        self.loader = DocumentLoader('data/documents')
        self.processor = TextProcessor()
        self.embeddings_manager = EmbeddingsManager(self.api_key)

        # Initialize system
        self.initialize_system()

    def initialize_system(self):
        # Load and process documents
        documents = self.loader.load_documents()
        self.chunks = []
        for doc in documents:
            self.chunks.extend(self.processor.split_into_chunks(doc))

        # Create embeddings
        self.embeddings = self.embeddings_manager.create_embeddings(self.chunks)

        # Initialize retrieval system
        self.retrieval_system = RetrievalSystem(self.chunks, self.embeddings)

    def answer_question(self, question: str) -> str:
        # Get question embedding
        question_embedding = self.embeddings_manager.create_embeddings([question])[0]

        # Get relevant chunks
        relevant_chunks = self.retrieval_system.find_similar_chunks(question_embedding)

        # Prepare context
        context = "\n".join([chunk[0] for chunk in relevant_chunks])

        # Create prompt
        prompt = f"""Context: {context}\n\nQuestion: {question}\n\nAnswer:"""

        # Get response from OpenAI
        response = openai.chat.completions.create(
            model="gpt-4-turbo-preview",
            messages=[
                {"role": "system", "content": "You are a helpful assistant. Use the provided context to answer the question."},
                {"role": "user", "content": prompt}
            ]
        )

        return response.choices[0].message.content

>

>將樣本文檔放在test.py>中。 然後，運行：

.txtdata/documents結論： test.py

>這提供了一個基本的抹布系統。 未來的改進可能包括增強的塊，嵌入緩存，錯誤處理，精緻的及時工程和矢量數據庫集成。 請記住要安全地管理OpenAI API密鑰並監視使用量。

# test.py
from src.rag_system import RAGSystem

# Initialize the RAG system
rag = RAGSystem()

# Ask a question
question = "What was the answer to the guardian’s riddle, and how did it help Kai?" #Replace with your question based on your documents
answer = rag.answer_question(question)
print(answer)

>

以上是使用Python和Openai構建您的第一個抹布系統的詳細內容。更多資訊請關注PHP中文網其他相關文章！