Pendekatan Praktikal untuk Pengekstrakan Maklumat Utama (Bahagian 1)

Hai, ini Mrzaizai2k lagi!

Dalam siri ini, saya ingin berkongsi pendekatan saya untuk menyelesaikan masalah pengekstrakan maklumat utama (KIE) daripada invois. Kami akan meneroka cara memanfaatkan model bahasa besar (LLM) seperti ChatGPT dan Qwen2 untuk pengekstrakan maklumat. Kemudian, kita akan menyelami penggunaan model OCR seperti PaddleOCR, model klasifikasi tangkapan sifar atau Llama 3.1 untuk pasca memproses keputusan.

— Sial, ini menarik!

Untuk meningkatkan keadaan, kami akan mengendalikan invois dalam sebarang format dan sebarang bahasa. Ya, betul — ini nyata!

Menganalisis Keperluan

Mari bayangkan anda perlu membina perkhidmatan yang mengekstrak semua maklumat yang berkaitan daripada sebarang jenis invois, dalam sebarang bahasa. Sesuatu seperti yang anda akan temui di tapak web sampel ini.

Berikut ialah contoh imej invois yang akan kami usahakan:

Pertimbangan Utama

Pertama, mari analisa keperluan secara terperinci. Ini akan membantu kami memutuskan susunan teknologi yang betul untuk sistem kami. Walaupun teknologi tertentu mungkin berfungsi dengan baik, teknologi tersebut mungkin tidak sesuai untuk setiap senario. Inilah perkara yang perlu kita utamakan, dari atas ke bawah:

1. Lancarkan sistem dengan cepat
2. Pastikan ketepatan
3. Jadikan ia berfungsi pada sumber pengiraan terhad
   • (cth., GPU RTX 3060 dengan 12 GB VRAM atau malah CPU)
4. Pastikan masa pemprosesan berpatutan
   • ~1 minit setiap invois pada CPU, ~10 saat pada GPU
5. Fokus pada mengekstrak butiran yang berguna dan penting sahaja

Memandangkan keperluan ini, kami tidak akan memperhalusi apa-apa. Sebaliknya, kami akan menggabungkan teknologi sedia ada dan menyusunnya bersama-sama untuk mendapatkan hasil dengan cepat dan tepat—untuk sebarang format dan bahasa.

Sebagai penanda aras, saya perhatikan tapak web sampel memproses invois dalam kira-kira 3-4 saat. Jadi menyasarkan 10 saat dalam sistem kami boleh dicapai sepenuhnya.

Format output hendaklah sepadan dengan yang digunakan pada tapak web sampel:

Chatgpt

Baiklah, mari bercakap tentang alat pertama: ChatGPT. Anda mungkin sudah tahu betapa mudahnya untuk digunakan. Jadi, kenapa perlu bersusah payah membaca blog ini? Nah, bagaimana jika saya memberitahu anda bahawa saya boleh membantu anda mengoptimumkan penggunaan token dan mempercepatkan pemprosesan? Tertarik lagi? Tunggu sahaja—saya akan terangkan caranya.

Pendekatan Asas

Berikut ialah coretan kod asas. (Nota: Kod ini mungkin tidak sempurna—ini lebih kepada idea daripada pelaksanaan yang tepat). Anda boleh menyemak kod penuh dalam repositori OCR Invois Berbilang Bahasa repo saya.

class OpenAIExtractor(BaseExtractor):
    def __init__(self, config_path: str = "config/config.yaml"):
        super().__init__(config_path)

        self.config = self.config['openai']
        self.model = self.config['model_name']
        self.temperature = self.config['temperature']
        self.max_tokens = self.config['max_tokens']

        self.OPENAI_API_KEY = os.getenv('OPENAI_API_KEY')
        from openai import OpenAI
        self.client = OpenAI(api_key=self.OPENAI_API_KEY)

    def _extract_invoice_llm(self, ocr_text, base64_image:str, invoice_template:str):
        response = self.client.chat.completions.create(
            model=self.model,
            messages=[
                {"role": "system", "content": """You are a helpful assistant that responds in JSON format with the invoice information in English. 
                                            Don't add any annotations there. Remember to close any bracket. Number, price and amount should be number, date should be convert to dd/mm/yyyy, 
                                            time should be convert to HH:mm:ss, currency should be 3 chracters like VND, USD, EUR"""},
                {"role": "user", "content": [
                    {"type": "text", "text": f"From the image of the bill and the text from OCR, extract the information. The ocr text is: {ocr_text} \n. Return the key names as in the template is a MUST. The invoice template: \n {invoice_template}"},
                    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{base64_image}"}}
                ]}
            ],
            temperature=self.temperature,
            max_tokens=self.max_tokens,
        )
        return response.choices[0].message.content

    def extract_json(self, text: str) -> dict:
        start_index = text.find('{')
        end_index = text.rfind('}') + 1
        json_string = text[start_index:end_index]
        json_string = json_string.replace('true', 'True').replace('false', 'False').replace('null', 'None')
        result = eval(json_string)
        return result

    @retry_on_failure(max_retries=3, delay=1.0)
    def extract_invoice(self, ocr_text, image: Union[str, np.ndarray], invoice_template:str) -> dict:
        base64_image = self.encode_image(image)
        invoice_info = self._extract_invoice_llm(ocr_text, base64_image, 
                                                 invoice_template=invoice_template)
        invoice_info = self.extract_json(invoice_info)
        return invoice_info

ok, jom tengok hasilnya

invoice {
    "invoice_info": {
        "amount": 32.0,
        "amount_change": 0,
        "amount_shipping": 0,
        "vatamount": 0,
        "amountexvat": 32.0,
        "currency": "EUR",
        "purchasedate": "28/06/2008",
        "purchasetime": "17:46:26",
        "vatitems": [
            {
                "amount": 32.0,
                "amount_excl_vat": 32.0,
                "amount_incl_vat": 32.0,
                "amount_incl_excl_vat_estimated": false,
                "percentage": 0,
                "code": ""
            }
        ],
        "vat_context": "",
        "lines": [
            {
                "description": "",
                "lineitems": [
                    {
                        "title": "Lunettes",
                        "description": "",
                        "amount": 22.0,
                        "amount_each": 22.0,
                        "amount_ex_vat": 22.0,
                        "vat_amount": 0,
                        "vat_percentage": 0,
                        "quantity": 1,
                        "unit_of_measurement": "",
                        "sku": "",
                        "vat_code": ""
                    },
                    {
                        "title": "Chapeau",
                        "description": "",
                        "amount": 10.0,
                        "amount_each": 10.0,
                        "amount_ex_vat": 10.0,
                        "vat_amount": 0,
                        "vat_percentage": 0,
                        "quantity": 1,
                        "unit_of_measurement": "",
                        "sku": "",
                        "vat_code": ""
                    }
                ]
            }
        ],
        "paymentmethod": "CB EMV",
        "payment_auth_code": "",
        "payment_card_number": "",
        "payment_card_account_number": "",
        "payment_card_bank": "",
        "payment_card_issuer": "",
        "payment_due_date": "",
        "terminal_number": "",
        "document_subject": "",
        "package_number": "",
        "invoice_number": "",
        "receipt_number": "000130",
        "shop_number": "",
        "transaction_number": "000148",
        "transaction_reference": "",
        "order_number": "",
        "table_number": "",
        "table_group": "",
        "merchant_name": "G\u00e9ant Casino",
        "merchant_id": "",
        "merchant_coc_number": "",
        "merchant_vat_number": "",
        "merchant_bank_account_number": "",
        "merchant_bank_account_number_bic": "",
        "merchant_chain_liability_bank_account_number": "",
        "merchant_chain_liability_amount": 0,
        "merchant_bank_domestic_account_number": "",
        "merchant_bank_domestic_bank_code": "",
        "merchant_website": "",
        "merchant_email": "",
        "merchant_address": "Annecy",
        "merchant_phone": "04.50.88.20.00",
        "customer_name": "",
        "customer_address": "",
        "customer_phone": "",
        "customer_website": "",
        "customer_vat_number": "",
        "customer_coc_number": "",
        "customer_bank_account_number": "",
        "customer_bank_account_number_bic": "",
        "customer_email": "",
        "document_language": ""
    }
}
Test_Openai_Invoice Took 0:00:11.15

Hasilnya agak kukuh, tetapi masa pemprosesan adalah masalah—ia melebihi had 10 saat kami. Anda juga mungkin mendapati bahawa output termasuk banyak medan kosong, yang bukan sahaja meningkatkan masa pemprosesan tetapi juga boleh memperkenalkan ralat dan menggunakan lebih banyak token—pada asasnya memerlukan lebih banyak wang.

Pendekatan Lanjutan

Nampaknya, kami hanya memerlukan sedikit perubahan untuk membetulkannya.

Cuma tambah ayat berikut pada gesaan anda:

"Hanya medan keluaran yang mempunyai nilai dan jangan kembalikan sebarang medan kosong."

Vila! Masalah selesai!

 invoice_info {
    "invoice_info": {
        "amount": 32,
        "currency": "EUR",
        "purchasedate": "28/06/2008",
        "purchasetime": "17:46:26",
        "lines": [
            {
                "description": "",
                "lineitems": [
                    {
                        "title": "LUNETTES",
                        "description": "",
                        "amount": 22,
                        "amount_each": 22,
                        "amount_ex_vat": 22,
                        "vat_amount": 0,
                        "vat_percentage": 0,
                        "quantity": 1,
                        "unit_of_measurement": "",
                        "sku": "",
                        "vat_code": ""
                    },
                    {
                        "title": "CHAPEAU",
                        "description": "",
                        "amount": 10,
                        "amount_each": 10,
                        "amount_ex_vat": 10,
                        "vat_amount": 0,
                        "vat_percentage": 0,
                        "quantity": 1,
                        "unit_of_measurement": "",
                        "sku": "",
                        "vat_code": ""
                    }
                ]
            }
        ],
        "invoice_number": "000130"
    }
}
Test_Openai_Invoice Took 0:00:05.79

Wah, sungguh penukar permainan! Kini keputusannya lebih pendek dan lebih tepat, dan masa pemprosesan telah menurun daripada 11.15 saat kepada hanya 5.79 saat. Dengan tweak ayat tunggal itu, kami telah mengurangkan kos dan masa pemprosesan sekitar 50%. Agak hebat, bukan?

Dalam kes ini, saya menggunakan GPT-4o-mini, yang berfungsi dengan baik, tetapi dalam pengalaman saya, Gemini Flash berprestasi lebih baik—lebih pantas dan percuma! Pasti berbaloi untuk dilihat.

Anda boleh mengoptimumkan lagi perkara dengan memendekkan templat, memfokuskan hanya pada medan yang paling penting berdasarkan keperluan khusus anda.

PaddleOCR

Hasilnya kelihatan agak bagus, tetapi masih terdapat beberapa medan yang hilang, seperti nombor telefon atau nama juruwang, yang juga ingin kami tangkap. Walaupun kami hanya boleh menggesa semula ChatGPT, bergantung semata-mata pada LLM boleh menjadi tidak dapat diramalkan—hasilnya boleh berbeza-beza mengikut setiap larian. Selain itu, templat gesaan agak panjang (memandangkan kami cuba mengekstrak semua maklumat yang mungkin untuk semua format), yang boleh menyebabkan ChatGPT "melupakan" butiran tertentu.

Di sinilah PaddleOCR masuk—ia meningkatkan keupayaan penglihatan LLM dengan menyediakan teks OCR yang tepat, membantu model memfokus pada perkara yang perlu diekstrak.

In my previous prompt, I used this structure:

{"type": "text", "text": f"From the image of the bill and the text from OCR, extract the information. The ocr text is: {ocr_text} \n.

Previously, I set ocr_text = '', but now we'll populate it with the output from PaddleOCR. Since I'm unsure of the specific language for now, I'll use English (as it's the most commonly supported). In the next part, I’ll guide you on detecting the language, so hang tight!

Here’s the updated code to integrate PaddleOCR:

ocr = PaddleOCR(lang='en', show_log=False, use_angle_cls=True, cls=True)
result = ocr.ocr(np.array(image))

This is the OCR output.

    "Geant Casino ANNECY BIENVENUE DANS NOTRE MAGASIN Caisse014 Date28/06/2008 VOTRE MAGASIN VOUS ACCUEILLE DU LUNDI AU SAMEDI DE 8H30 A21H00 TEL.04.50.88.20.00 LUNETTES 22.00E CHAPEAU 10.00E =TOTAL2) 32.00E CB EMV 32.00E Si vous aviez la carte fidelite, vous auriez cumule 11SMILES Caissier:000148/Heure:17:46:26 Numero de ticket :000130 Rapidite,confort d'achat budget maitrise.. Scan' Express vous attend!! Merci de votre visite A bientot"

As you can see, the results are pretty good. In this case, the invoice is in French, which looks similar to English, so the output is decent. However, if we were dealing with languages like Japanese or Chinese, the results wouldn't be as accurate.

Now, let’s see what happens when we combine the OCR output with ChatGPT.

 invoice_info {
    "invoice_info": {
        "amount": 32,
        "currency": "EUR",
        "purchasedate": "28/06/2008",
        "purchasetime": "17:46:26",
        "lines": [
            {
                "description": "",
                "lineitems": [
                    {
                        "title": "LUNETTES",
                        "description": "",
                        "amount": 22,
                        "amount_each": 22,
                        "amount_ex_vat": 22,
                        "vat_amount": 0,
                        "vat_percentage": 0,
                        "quantity": 1,
                        "unit_of_measurement": "",
                        "sku": "",
                        "vat_code": ""
                    },
                    {
                        "title": "CHAPEAU",
                        "description": "",
                        "amount": 10,
                        "amount_each": 10,
                        "amount_ex_vat": 10,
                        "vat_amount": 0,
                        "vat_percentage": 0,
                        "quantity": 1,
                        "unit_of_measurement": "",
                        "sku": "",
                        "vat_code": ""
                    }
                ]
            }
        ],
        "paymentmethod": "CB EMV",
        "receipt_number": "000130",
        "transaction_number": "000130",
        "merchant_name": "G\u00e9ant Casino",
        "customer_email": "",
        "customer_name": "",
        "customer_address": "",
        "customer_phone": ""
    }
}
Test_Openai_Invoice Took 0:00:06.78

Awesome! It uses a few more tokens and takes slightly longer, but it returns additional fields like payment_method, receipt_number, and cashier. That’s a fair trade-off and totally acceptable!

Language Detection

Right now, we’re facing two major challenges. First, PaddleOCR cannot automatically detect the language, which significantly affects the OCR output, and ultimately impacts the entire result. Second, most LLMs perform best with English, so if the input is in another language, the quality of the results decreases.

To demonstrate, I’ll use a challenging example.

Here’s a Japanese invoice:

Let’s see what happens if we fail to auto-detect the language and use lang='en' to extract OCR on this Japanese invoice.

The result

'TEL045-752-6131 E TOP&CIubQJMB-FJ 2003 20130902 LNo.0102 No0073 0011319-2x198 396 00327111 238 000805 VR-E--E 298 003276 9 -435 298 001093 398 000335 138 000112 7 2x158 316 A000191 92 29 t 2.111 100) 10.001 10.001 7.890'

As you can see, the result is pretty bad.

Now, let’s detect the language using a zero-shot classification model. In this case, I’m using "facebook/metaclip-b32-400m". This is one of the best ways to detect around 80 languages supported by PaddleOCR without needing fine-tuning while still maintaining accuracy.

def initialize_language_detector(self):
        # Initialize the zero-shot image classification model
        self.image_classifier = pipeline(task="zero-shot-image-classification", 
                                         model="facebook/metaclip-b32-400m", 
                                         device=self.device,
                                         batch_size=8)

def _get_lang(self, image: Image.Image) -> str:
        # Define candidate labels for language classification
        candidate_labels = [f"language {key}" for key in self.language_dict]

        # Perform inference to classify the language
        outputs = self.image_classifier(image, candidate_labels=candidate_labels)
        outputs = [{"score": round(output["score"], 4), "label": output["label"] } for output in outputs]

        # Extract the language with the highest score
        language_names = [entry['label'].replace('language ', '') for entry in outputs]
        scores = [entry['score'] for entry in outputs]
        abbreviations = [self.language_dict.get(language) for language in language_names]

        first_abbreviation = abbreviations[0]
        lang = 'en'  # Default to English

        if scores[0] > self.language_thresh:
            lang = first_abbreviation
        print("The source language", abbreviations)
        return lang

Let's see the result

Recognized Text: 
{'ori_text': '根岸 東急ストア TEL 045-752-6131 領収証 [TOP2C!UbO J3カード」 クレヅッ 卜でのお支払なら 200円で3ボイン卜 お得なカード! 是非こ入会下さい。 2013年09月02日(月) レジNO. 0102 NOO07さ と う 001131 スダフエウ卜チーネ 23 単198 1396 003271 オインイ年 ユウ10 4238 000805 ソマ一ク スモー一クサーモン 1298 003276 タカナン ナマクリーム35 1298 001093 ヌテラ スフレクト 1398 000335 バナサ 138 000112 アボト 2つ 単158 1316 A000191 タマネキ 429 合計 2,111 (内消費税等 100 現金 10001 お預り合計 110 001 お釣り 7 890', 
'ori_language': 'ja', 
'text': 'Negishi Tokyu Store TEL 045-752-6131 Receipt [TOP2C!UbO J3 Card] If you pay with a credit card, you can get 3 points for 200 yen.A great value card!Please join us. Monday, September 2, 2013 Cashier No. 0102 NOO07 Satou 001131 Sudafue Bucine 23 Single 198 1396 003271 Oinyen Yu 10 4238 000805 Soma Iku Smo Iku Salmon 1298 003276 Takanan Nama Cream 35 1 298 001093 Nutella Sprect 1398 000335 Banasa 138 000112 Aboto 2 AA 158 1316 A000191 Eggplant 429 Total 2,111 (including consumption tax, etc. 100 Cash 10001 Total deposited 110 001 Change 7 890', 
'language': 'en',}

The results are much better now! I also translated the original Japanese into English. With this approach, the output will significantly improve for other languages as well.

Summary

In this blog, we explored how to extract key information from invoices by combining LLMs and OCR, while also optimizing processing time, minimizing token usage, and improving multilingual support. By incorporating PaddleOCR and a zero-shot language detection model, we boosted both accuracy and reliability across different formats and languages. I hope these examples help you grasp the full process, from initial concept to final implementation.

Reference:

Mrzaizai2k - Multilanguage invoice ocr

More

If you’d like to learn more, be sure to check out my other posts and give me a like! It would mean a lot to me. Thank you.

• Real-Time Data Processing with MongoDB Change Streams and Python
• Replay Attack: Let’s Learn
• Reasons to Write

Atas ialah kandungan terperinci Pendekatan Praktikal untuk Pengekstrakan Maklumat Utama (Bahagian 1). Untuk maklumat lanjut, sila ikut artikel berkaitan lain di laman web China PHP!