Perkhidmatan Terjemahan Dokumen menggunakan Streamlit & AWS Translator

pengenalan:

DocuTranslator, sistem terjemahan dokumen, dibina dalam AWS dan dibangunkan oleh rangka kerja aplikasi Streamlit. Aplikasi ini membolehkan pengguna akhir menterjemah dokumen dalam bahasa pilihan mereka yang ingin mereka muat naik. Ia menyediakan kemungkinan untuk menterjemah dalam pelbagai bahasa mengikut kehendak pengguna, yang benar-benar membantu pengguna memahami kandungan dengan cara yang selesa.

latar belakang:

Niat projek ini adalah untuk menyediakan antara muka aplikasi yang mesra pengguna dan mudah untuk memenuhi proses terjemahan semudah yang pengguna jangkakan. Dalam sistem ini, tiada siapa yang perlu menterjemah dokumen dengan memasuki perkhidmatan AWS Translate, sebaliknya pengguna akhir boleh terus mengakses titik akhir aplikasi dan memenuhi keperluan.

Rajah Seni Bina Aras Tinggi:

Bagaimana Ini Berfungsi:

• Pengguna akhir dibenarkan mengakses aplikasi melalui pengimbang beban aplikasi.
• Sebaik sahaja antara muka aplikasi dibuka, pengguna akan memuat naik fail yang diperlukan untuk diterjemahkan dan bahasa untuk diterjemahkan.
• Selepas menyerahkan butiran ini, fail akan dimuat naik ke baldi S3 sumber yang disebutkan yang mencetuskan fungsi lambda untuk menyambung dengan perkhidmatan AWS Translator.
• Setelah dokumen diterjemahkan siap, akan dimuat naik ke baldi S3 destinasi.
• Selepas itu, pengguna akhir boleh memuat turun dokumen terjemahan daripada portal aplikasi Streamlit.

Seni Bina Teknikal:

Seni bina di atas menunjukkan di bawah perkara utama -

• Kod aplikasi telah disimpan dalam bekas dan disimpan ke repositori ECR.
• Seperti reka bentuk di atas, gugusan ECS telah disediakan yang mewujudkan dua tugasan yang menarik imej aplikasi daripada repositori ECR.
• Kedua-dua tugasan dilancarkan di atas EC2 sebagai jenis pelancaran. Kedua-dua EC2 dilancarkan dalam subnet peribadi di zon ketersediaan us-east-1a dan us-east-1b.
• Sistem fail EFS dicipta untuk berkongsi kod aplikasi antara dua kejadian EC2 asas. Dua titik pelekap dibuat dalam dua zon ketersediaan (us-east-1a dan us-east-1b).
• Dua subnet awam dikonfigurasikan di hadapan subnet peribadi dan get laluan NAT disediakan dalam subnet awam dalam zon ketersediaan us-east-1a.
• Pengimbang beban aplikasi telah dikonfigurasikan di hadapan subnet peribadi yang mengagihkan trafik merentas dua subnet awam di port 80 kumpulan keselamatan pengimbang beban aplikasi(ALB SG).
• Dua kejadian EC2 dikonfigurasikan dalam dua kumpulan sasaran berbeza dengan kumpulan keselamatan EC2 yang sama(Streamlit_SG) yang menerima trafik di port 16347 daripada pengimbang beban aplikasi.
• Terdapat pemetaan port yang dikonfigurasikan antara port 16347 dalam keadaan EC2 dan port 8501 pada bekas ECS. Sebaik sahaja trafik melanda port 16347 kumpulan keselamatan EC2, akan diubah hala ke port 8501 pada tahap kontena ECS.

Bagaimana Data Disimpan?

Di sini, kami telah menggunakan laluan perkongsian EFS untuk berkongsi fail aplikasi yang sama antara dua kejadian EC2 asas. Kami telah mencipta titik lekap /streamlit_appfiles di dalam tika EC2 dan dipasang dengan bahagian EFS. Pendekatan ini akan membantu dalam berkongsi kandungan yang sama merentas dua pelayan berbeza. Selepas itu, niat kami adalah untuk mencipta kandungan aplikasi yang sama replika ke direktori kerja kontena iaitu /meniruskan. Untuk itu kami telah menggunakan pelekap bind supaya apa-apa perubahan yang akan dibuat pada kod aplikasi pada tahap EC2, akan direplikasi kepada bekas juga. Kita perlu mengehadkan replikasi dwi-arah yang mengatakan jika sesiapa tersilap menukar kod dari dalam bekas, ia tidak seharusnya meniru kepada tahap hos EC2, justeru di dalam direktori kerja kontena telah dibuat sebagai sistem fail baca sahaja.

Konfigurasi dan Kelantangan Bekas ECS:

Tatarajah EC2 yang mendasari:
Jenis Contoh: t2.medium
Jenis rangkaian: Subnet Peribadi

Konfigurasi Bekas:
Imej:
Mod Rangkaian: Lalai
Pelabuhan Hos: 16347
Pelabuhan Kontena: 8501
CPU Tugasan: 2 vCPU (2048 unit)
Memori Tugasan: 2.5 GB (2560 MiB)

Konfigurasi Kelantangan:
Nama Jilid: streamlit-volume
Laluan Sumber: /streamlit_appfiles
Laluan Kontena: /streamlit
Sistem Fail Baca Sahaja: YA

Rujukan Definisi Tugas:

{
    "taskDefinitionArn": "arn:aws:ecs:us-east-1:<account-id>:task-definition/Streamlit_TDF-1:5",
    "containerDefinitions": [
        {
            "name": "streamlit",
            "image": "<account-id>.dkr.ecr.us-east-1.amazonaws.com/anirban:latest",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "streamlit-8501-tcp",
                    "containerPort": 8501,
                    "hostPort": 16347,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environment": [],
            "environmentFiles": [],
            "mountPoints": [
                {
                    "sourceVolume": "streamlit-volume",
                    "containerPath": "/streamlit",
                    "readOnly": true
                }
            ],
            "volumesFrom": [],
            "ulimits": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/ecs/Streamlit_TDF-1",
                    "mode": "non-blocking",
                    "awslogs-create-group": "true",
                    "max-buffer-size": "25m",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "ecs"
                },
                "secretOptions": []
            },
            "systemControls": []
        }
    ],
    "family": "Streamlit_TDF-1",
    "taskRoleArn": "arn:aws:iam::<account-id>:role/ecsTaskExecutionRole",
    "executionRoleArn": "arn:aws:iam::<account-id>:role/ecsTaskExecutionRole",
    "revision": 5,
    "volumes": [
        {
            "name": "streamlit-volume",
            "host": {
                "sourcePath": "/streamlit_appfiles"
            }
        }
    ],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
        },
        {
            "name": "ecs.capability.execution-role-awslogs"
        },
        {
            "name": "com.amazonaws.ecs.capability.ecr-auth"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.28"
        },
        {
            "name": "com.amazonaws.ecs.capability.task-iam-role"
        },
        {
            "name": "ecs.capability.execution-role-ecr-pull"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2"
    ],
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "2048",
    "memory": "2560",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    },
    "registeredAt": "2024-11-09T05:59:47.534Z",
    "registeredBy": "arn:aws:iam::<account-id>:root",
    "tags": []
}
</account-id></account-id></account-id></account-id></account-id>

Membangunkan Kod Aplikasi dan Mencipta Imej Docker:

app.py

import streamlit as st
import boto3
import os
import time
from pathlib import Path

s3 = boto3.client('s3', region_name='us-east-1')
tran = boto3.client('translate', region_name='us-east-1')
lam = boto3.client('lambda', region_name='us-east-1')


# Function to list S3 buckets
def listbuckets():
    list_bucket = s3.list_buckets()
    bucket_name = tuple([it["Name"] for it in list_bucket["Buckets"]])
    return bucket_name

# Upload object to S3 bucket
def upload_to_s3bucket(file_path, selected_bucket, file_name):
    s3.upload_file(file_path, selected_bucket, file_name)

def list_language():
    response = tran.list_languages()
    list_of_langs = [i["LanguageName"] for i in response["Languages"]]
    return list_of_langs

def wait_for_s3obj(dest_selected_bucket, file_name):
    while True:
        try:
            get_obj = s3.get_object(Bucket=dest_selected_bucket, Key=f'Translated-{file_name}.txt')
            obj_exist = 'true' if get_obj['Body'] else 'false'
            return obj_exist
        except s3.exceptions.ClientError as e:
            if e.response['Error']['Code'] == "404":
                print(f"File '{file_name}' not found. Checking again in 3 seconds...")
                time.sleep(3)

def download(dest_selected_bucket, file_name, file_path):
     s3.download_file(dest_selected_bucket,f'Translated-{file_name}.txt', f'{file_path}/download/Translated-{file_name}.txt')
     with open(f"{file_path}/download/Translated-{file_name}.txt", "r") as file:
       st.download_button(
             label="Download",
             data=file,
             file_name=f"{file_name}.txt"
       )

def streamlit_application():
    # Give a header
    st.header("Document Translator", divider=True)
    # Widgets to upload a file
    uploaded_files = st.file_uploader("Choose a PDF file", accept_multiple_files=True, type="pdf")
    # # upload a file
    file_name = uploaded_files[0].name.replace(' ', '_') if uploaded_files else None
    # Folder path
    file_path = '/tmp'
    # Select the bucket from drop down
    selected_bucket = st.selectbox("Choose the S3 Bucket to upload file :", listbuckets())
    dest_selected_bucket = st.selectbox("Choose the S3 Bucket to download file :", listbuckets())
    selected_language = st.selectbox("Choose the Language :", list_language())
    # Create a button
    click = st.button("Upload", type="primary")
    if click == True:
        if file_name:
            with open(f'{file_path}/{file_name}', mode='wb') as w:
                w.write(uploaded_files[0].getvalue())
        # Set the selected language to the environment variable of lambda function
        lambda_env1 = lam.update_function_configuration(FunctionName='TriggerFunctionFromS3', Environment={'Variables': {'UserInputLanguage': selected_language, 'DestinationBucket': dest_selected_bucket, 'TranslatedFileName': file_name}})
        # Upload the file to S3 bucket:
        upload_to_s3bucket(f'{file_path}/{file_name}', selected_bucket, file_name)
        if s3.get_object(Bucket=selected_bucket, Key=file_name):
            st.success("File uploaded successfully", icon="✅")
            output = wait_for_s3obj(dest_selected_bucket, file_name)
            if output:
              download(dest_selected_bucket, file_name, file_path)
        else:
            st.error("File upload failed", icon="?")


streamlit_application()

tentang.py

import streamlit as st

## Write the description of application
st.header("About")
about = '''
Welcome to the File Uploader Application!

This application is designed to make uploading PDF documents simple and efficient. With just a few clicks, users can upload their documents securely to an Amazon S3 bucket for storage. Here’s a quick overview
of what this app does:

**Key Features:**
- **Easy Upload:** Users can quickly upload PDF documents by selecting the file and clicking the 'Upload' button.
- **Seamless Integration with AWS S3:** Once the document is uploaded, it is stored securely in a designated S3 bucket, ensuring reliable and scalable cloud storage.
- **User-Friendly Interface:** Built using Streamlit, the interface is clean, intuitive, and accessible to all users, making the uploading process straightforward.

**How it Works:**
1. **Select a PDF Document:** Users can browse and select any PDF document from their local system.
2. **Upload the Document:** Clicking the ‘Upload’ button triggers the process of securely uploading the selected document to an AWS S3 bucket.
3. **Success Notification:** After a successful upload, users will receive a confirmation message that their document has been stored in the cloud.
This application offers a streamlined way to store documents on the cloud, reducing the hassle of manual file management. Whether you're an individual or a business, this tool helps you organize and store your
 files with ease and security.
You can further customize this page by adding technical details, usage guidelines, or security measures as per your application's specifications.'''

st.markdown(about)

navigation.py

import streamlit as st

pg = st.navigation([
    st.Page("app.py", title="DocuTranslator", icon="?"),
    st.Page("about.py", title="About", icon="?")
], position="sidebar")

pg.run()

Fail Docker:

FROM python:3.9-slim
WORKDIR /streamlit
COPY requirements.txt /streamlit/requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
RUN mkdir /tmp/download
COPY . /streamlit
EXPOSE 8501
CMD ["streamlit", "run", "navigation.py", "--server.port=8501", "--server.headless=true"]

Fail Docker akan mencipta imej dengan membungkus semua fail konfigurasi aplikasi di atas dan kemudian ia ditolak ke repositori ECR. Docker Hub juga boleh digunakan untuk menyimpan imej.

Pengimbangan Beban

Dalam seni bina, tika aplikasi sepatutnya dibuat dalam subnet peribadi dan pengimbang beban sepatutnya dibuat untuk mengurangkan beban trafik masuk ke tika EC2 peribadi.
Memandangkan terdapat dua hos EC2 asas yang tersedia untuk bekas hos, jadi pengimbangan beban dikonfigurasikan merentas dua hos EC2 untuk mengagihkan trafik masuk. Dua kumpulan sasaran berbeza dicipta untuk meletakkan dua kejadian EC2 dalam setiap satu dengan wajaran 50%.

Pengimbang beban menerima trafik masuk di port 80 dan kemudian meneruskan ke belakang kejadian EC2 di port 16347 dan itu juga dihantar ke bekas ECS yang sepadan.

Fungsi Lambda:

Terdapat fungsi lambda yang dikonfigurasikan untuk mengambil baldi sumber sebagai input untuk memuat turun fail pdf dari sana dan mengekstrak kandungannya, kemudian ia menterjemah kandungan daripada bahasa semasa kepada bahasa sasaran yang disediakan pengguna dan mencipta fail teks untuk dimuat naik ke destinasi S3 baldi.

{
    "taskDefinitionArn": "arn:aws:ecs:us-east-1:<account-id>:task-definition/Streamlit_TDF-1:5",
    "containerDefinitions": [
        {
            "name": "streamlit",
            "image": "<account-id>.dkr.ecr.us-east-1.amazonaws.com/anirban:latest",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "streamlit-8501-tcp",
                    "containerPort": 8501,
                    "hostPort": 16347,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environment": [],
            "environmentFiles": [],
            "mountPoints": [
                {
                    "sourceVolume": "streamlit-volume",
                    "containerPath": "/streamlit",
                    "readOnly": true
                }
            ],
            "volumesFrom": [],
            "ulimits": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/ecs/Streamlit_TDF-1",
                    "mode": "non-blocking",
                    "awslogs-create-group": "true",
                    "max-buffer-size": "25m",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "ecs"
                },
                "secretOptions": []
            },
            "systemControls": []
        }
    ],
    "family": "Streamlit_TDF-1",
    "taskRoleArn": "arn:aws:iam::<account-id>:role/ecsTaskExecutionRole",
    "executionRoleArn": "arn:aws:iam::<account-id>:role/ecsTaskExecutionRole",
    "revision": 5,
    "volumes": [
        {
            "name": "streamlit-volume",
            "host": {
                "sourcePath": "/streamlit_appfiles"
            }
        }
    ],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
        },
        {
            "name": "ecs.capability.execution-role-awslogs"
        },
        {
            "name": "com.amazonaws.ecs.capability.ecr-auth"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.28"
        },
        {
            "name": "com.amazonaws.ecs.capability.task-iam-role"
        },
        {
            "name": "ecs.capability.execution-role-ecr-pull"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2"
    ],
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "2048",
    "memory": "2560",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    },
    "registeredAt": "2024-11-09T05:59:47.534Z",
    "registeredBy": "arn:aws:iam::<account-id>:root",
    "tags": []
}
</account-id></account-id></account-id></account-id></account-id>

Ujian Aplikasi:

Buka url pengimbang beban aplikasi "ALB-747339710.us-east-1.elb.amazonaws.com" untuk membuka aplikasi web. Semak imbas mana-mana fail pdf, pastikan kedua-dua sumber "fileuploadbucket-hwirio984092jjs" dan baldi destinasi "translatedfileuploadbucket-kh939809kjkfjsekfl" sebagaimana adanya, kerana dalam kod lambda, ia telah dikodkan keras baldi adalah seperti yang dinyatakan di atas. Pilih bahasa yang anda mahu dokumen itu diterjemahkan dan klik pada muat naik. Sebaik sahaja ia diklik, program aplikasi akan mula mengundi baldi S3 destinasi untuk mengetahui sama ada fail yang diterjemahkan telah dimuat naik. Jika ia menemui fail yang tepat, maka pilihan baharu "Muat turun" akan kelihatan untuk memuat turun fail dari baldi S3 destinasi.

Pautan Permohonan: http://alb-747339710.us-east-1.elb.amazonaws.com/

Kandungan Sebenar:

import streamlit as st
import boto3
import os
import time
from pathlib import Path

s3 = boto3.client('s3', region_name='us-east-1')
tran = boto3.client('translate', region_name='us-east-1')
lam = boto3.client('lambda', region_name='us-east-1')


# Function to list S3 buckets
def listbuckets():
    list_bucket = s3.list_buckets()
    bucket_name = tuple([it["Name"] for it in list_bucket["Buckets"]])
    return bucket_name

# Upload object to S3 bucket
def upload_to_s3bucket(file_path, selected_bucket, file_name):
    s3.upload_file(file_path, selected_bucket, file_name)

def list_language():
    response = tran.list_languages()
    list_of_langs = [i["LanguageName"] for i in response["Languages"]]
    return list_of_langs

def wait_for_s3obj(dest_selected_bucket, file_name):
    while True:
        try:
            get_obj = s3.get_object(Bucket=dest_selected_bucket, Key=f'Translated-{file_name}.txt')
            obj_exist = 'true' if get_obj['Body'] else 'false'
            return obj_exist
        except s3.exceptions.ClientError as e:
            if e.response['Error']['Code'] == "404":
                print(f"File '{file_name}' not found. Checking again in 3 seconds...")
                time.sleep(3)

def download(dest_selected_bucket, file_name, file_path):
     s3.download_file(dest_selected_bucket,f'Translated-{file_name}.txt', f'{file_path}/download/Translated-{file_name}.txt')
     with open(f"{file_path}/download/Translated-{file_name}.txt", "r") as file:
       st.download_button(
             label="Download",
             data=file,
             file_name=f"{file_name}.txt"
       )

def streamlit_application():
    # Give a header
    st.header("Document Translator", divider=True)
    # Widgets to upload a file
    uploaded_files = st.file_uploader("Choose a PDF file", accept_multiple_files=True, type="pdf")
    # # upload a file
    file_name = uploaded_files[0].name.replace(' ', '_') if uploaded_files else None
    # Folder path
    file_path = '/tmp'
    # Select the bucket from drop down
    selected_bucket = st.selectbox("Choose the S3 Bucket to upload file :", listbuckets())
    dest_selected_bucket = st.selectbox("Choose the S3 Bucket to download file :", listbuckets())
    selected_language = st.selectbox("Choose the Language :", list_language())
    # Create a button
    click = st.button("Upload", type="primary")
    if click == True:
        if file_name:
            with open(f'{file_path}/{file_name}', mode='wb') as w:
                w.write(uploaded_files[0].getvalue())
        # Set the selected language to the environment variable of lambda function
        lambda_env1 = lam.update_function_configuration(FunctionName='TriggerFunctionFromS3', Environment={'Variables': {'UserInputLanguage': selected_language, 'DestinationBucket': dest_selected_bucket, 'TranslatedFileName': file_name}})
        # Upload the file to S3 bucket:
        upload_to_s3bucket(f'{file_path}/{file_name}', selected_bucket, file_name)
        if s3.get_object(Bucket=selected_bucket, Key=file_name):
            st.success("File uploaded successfully", icon="✅")
            output = wait_for_s3obj(dest_selected_bucket, file_name)
            if output:
              download(dest_selected_bucket, file_name, file_path)
        else:
            st.error("File upload failed", icon="?")


streamlit_application()

Kandungan Terjemahan (dalam bahasa Perancis Kanada)

import streamlit as st

## Write the description of application
st.header("About")
about = '''
Welcome to the File Uploader Application!

This application is designed to make uploading PDF documents simple and efficient. With just a few clicks, users can upload their documents securely to an Amazon S3 bucket for storage. Here’s a quick overview
of what this app does:

**Key Features:**
- **Easy Upload:** Users can quickly upload PDF documents by selecting the file and clicking the 'Upload' button.
- **Seamless Integration with AWS S3:** Once the document is uploaded, it is stored securely in a designated S3 bucket, ensuring reliable and scalable cloud storage.
- **User-Friendly Interface:** Built using Streamlit, the interface is clean, intuitive, and accessible to all users, making the uploading process straightforward.

**How it Works:**
1. **Select a PDF Document:** Users can browse and select any PDF document from their local system.
2. **Upload the Document:** Clicking the ‘Upload’ button triggers the process of securely uploading the selected document to an AWS S3 bucket.
3. **Success Notification:** After a successful upload, users will receive a confirmation message that their document has been stored in the cloud.
This application offers a streamlined way to store documents on the cloud, reducing the hassle of manual file management. Whether you're an individual or a business, this tool helps you organize and store your
 files with ease and security.
You can further customize this page by adding technical details, usage guidelines, or security measures as per your application's specifications.'''

st.markdown(about)

Kesimpulan:

Artikel ini telah menunjukkan kepada kita bagaimana proses terjemahan dokumen boleh semudah yang kita bayangkan di mana pengguna akhir perlu mengklik beberapa pilihan untuk memilih maklumat yang diperlukan dan mendapatkan output yang diingini dalam masa beberapa saat tanpa memikirkan konfigurasi. Buat masa ini, kami telah memasukkan satu ciri untuk menterjemah dokumen pdf, tetapi kemudian, kami akan menyelidik lebih lanjut mengenai perkara ini untuk mempunyai pelbagai fungsi dalam satu aplikasi dengan mempunyai beberapa ciri menarik.

Atas ialah kandungan terperinci Perkhidmatan Terjemahan Dokumen menggunakan Streamlit & AWS Translator. Untuk maklumat lanjut, sila ikut artikel berkaitan lain di laman web China PHP!