search
HomeTechnology peripheralsAIiFLYTEK ICDAR 2023: Image and text recognition achieves greater glory again, winning four championships

ICDAR 2023 (International Conference on Document Analysis and Recognition), as one of the most important international conferences in the field of document image analysis and recognition, has recently received exciting news:

iFlytek Research Institute of HKUST and the University of Science and Technology of China The National Engineering Research Center for Speech and Language Information Processing (hereinafter referred to as the Research Center) won four championships in three competitions: multi-line formula recognition, document information location and extraction, and structured text information extraction.

MLHMETop: Focus on "multi-line writing", breakthrough in complexity

MLHME (multi-line formula recognition competition) test input After an image containing a handwritten mathematical formula, the algorithm outputs the accuracy of the corresponding LaTex string. It is worth mentioning that compared with previous mathematical formula recognition competitions, this competition has set "multi-line writing" as the main challenge for the first time in the industry. And unlike the previous formulas for recognizing scanned and online handwriting, this time it will recognize photographed handwriting. Mainly multi-line formulas.

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

The image and text recognition team of iFlytek Research Institute won the championship with a score of 67.9%, and far exceeded other participating teams in the main evaluation indicator-formula recall rate

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

The recall rate of the formula corresponds to the Submit Results in the list

Multi-line formulas are more complex than single-line structures, and the same character appears more often in the formula The size will also change when it appears. At the same time, the data set used in the competition comes from real scenes, and the handwritten formula pictures taken have problems such as low quality, background interference, text interference, smearing and annotation interference. These factors make the game more difficult.

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

The formula structure is complex and takes up multiple lines

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

The picture quality is not high and correction interference

Aiming at the problem that the formula has a complex structure and occupies multiple lines, the team uses Conv2former with a large convolution kernel as the encoder structure, which expands the field of view of the model and better captures the structural characteristics of multi-line formulas; innovatively proposes structured sequence decoding based on transformer The SSD processor explicitly models the hierarchical relationships within multi-line formulas in a refined manner, which greatly improves the generalization of complex structures and better models structured semantics.

In response to the character ambiguity problem caused by picture quality problems, the team innovatively proposed a semantic-enhanced decoder training algorithm. Through joint training of semantics and vision, the decoder has intrinsic domain knowledge. When characters are difficult to identify, the model can adaptively use domain knowledge to make inferences and give the most reasonable recognition results.

In response to the problem of large changes in character size, the team proposed an adaptive character scale estimation algorithm and a multi-scale fusion decoding strategy, which greatly improved the model's robustness to changes in character size.

DocILEThe crown: "Pick one in the line", the two-track competition of document information positioning and extraction topped the list

DocILE (Document Information Location and Extraction Competition) evaluates the performance of machine learning methods in locating, extracting and identifying line items of key information in semi-structured business documents.

The competition is divided into two track tasks: KILE and LIR. The KILE task needs to locate the key information location of predefined categories in the document. On this basis, the LIR task further groups each key information into different line items (Line Item), such as a single object (quantity, price) in a row in the table. )wait. iFlytek and the Research Center finally won the championship in two tracks

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军KILE track ranking

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

LIR track ranking

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

The left is the KILE track description, the right is the LIR track description

It can be seen from the task icon given by the official event that the information to be extracted from the document The variety is very diverse. Among them, the KILE task not only needs to extract key information of predefined categories, but also obtain the specific location of the key information; in the LIR task, a line item may have multiple lines of text in a single table. In addition, there are many types of information in the data set of this competition and the document formats are complex and diverse, which greatly increases the challenge.

The joint team proposed two technological innovation plans at the algorithm level:

In the pre-training stage, we designed a document filter based on OCR quality by extracting 2.74 million pages of document images from unannotated documents provided by the organizer. Then, we use a pre-trained language model to obtain the semantic representation of each text line in the document, and use the masked sentence representation recovery task, pre-trained under different Top-K configurations (attention span of the document in the GraphDoc model A hyperparameter)

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

#In the data set fine-tuning stage, the team used the pre-trained GraphDoc to extract the multi-modal representation of the text box and perform classification operations. On the basis of the classification results, multi-modal representations are sent to the low-level attention fusion module for instance aggregation. Based on the instance aggregation, the high-level attention fusion module is used to realize the aggregation of line item instances. The proposed attention fusion The modules have the same structure but do not share parameters with each other. They can be used for both KILE and LIR tasks with good results.

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

SVRDThe crown: No. 1 in zero-sample ticket structured information extraction task, pre-training model test

SVRD (Structured Text Information Extraction) competition is divided into 4 track sub-tasks. iFlytek and the research center are competing in the very difficult zero-sample structured information extraction sub-track (Task3: E2E zero-sample structured text extraction). Won first place

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

Ranking order

In the context of the official designation of key elements that need to be extracted for different types of invoices, this track requires participating teams to use model output The corresponding content of these key elements in the picture, "zero sample" means that the invoice types of the training set and the test set have no overlap; the end-to-end prediction accuracy of the track test model, the weighted average of score1 and score2 is used as the final evaluation index .

For pre-training models, zero samples put forward higher requirements. At the same time, there are various invoice formats used in the competition, and the names of the ride stops, departure times and other elements in each format are different. In addition, invoice photos have problems such as background interference, reflection, and text overlap, which further increases the difficulty of identification and extraction.

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

Invoices in different formats

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

The invoice is interfered by the striped background

The team initially adopted a copy-generate dual-branch decoding strategy to carry out the feature extraction model. When the confidence of the front-end OCR result is high, copy it directly OCR results; when the confidence level of the OCR results is low, new prediction results are generated to alleviate the recognition errors introduced by the front-end OCR model

In addition, the team also extracts sentence-level graphdoc features based on the OCR results As input to the feature extraction model, this feature integrates multi-modal features of image, text, location, and layout, and has a stronger feature representation than single-modal plain text input.

On this basis, the team also combined multiple element extraction models such as UniLM, LiLT, and DocPrompt to further improve the final element extraction effect and enable it to show better performance in different scenarios and languages. Advantages

Education, finance, medical care, etc. have been implemented to help large models improve multi-modal capabilities

This time I chose relevant events of ICDAR 2023 for the challenge, from iFlytek's real-life scenario needs in actual business; event-related technologies have also penetrated into education, finance, medical care, justice, intelligent hardware and other fields, empowering multiple businesses and products.

In the field of education, the technical ability of handwritten formula recognition is frequently used, and machines can provide accurate identification, judgment and correction. For example, the personalized precision learning and AI diagnosis in iFlytek's AI learning machine; the "iFlytek Smart Window" teaching large screen used by teachers in class, and students' personalized learning manuals, etc., have all achieved great results;

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

Not long ago, the Spark Scientific Research Assistant was released on the iFlytek Global 1024 Developer Festival main forum. One of the three core functions of paper reading can realize intelligent interpretation of papers and quickly answer related questions. . Subsequently, on the basis of high-precision formula recognition, the effect of organic chemical structural formulas, graphics, icons, flow charts, tables and other structured scene recognition will be advanced. This function will also better help scientific researchers improve efficiency;

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

Document information positioning and extraction technology is widely used in the financial field, such as contract element extraction and review, bank bill element extraction, marketing content consumer protection review and other scenarios. These technologies can realize functions such as data analysis, information extraction and comparison review of documents or files, and help business data to be quickly entered, extracted and compared, thereby improving the efficiency of the review process and reducing costs

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

The personal AI health assistant released on this 1024 main forum is iFlytek Xiaoyi. It can not only scan checklists and test orders and give analysis and suggestions, but it can also scan pill boxes and make further inquiries and provide auxiliary medication suggestions. For physical examination reports, users can take photos and upload them, and iFlytek Xiaoyi can identify key information, comprehensively interpret abnormal indicators, proactively inquire and provide more help. This function relies on the support of document information positioning and extraction technology

科大讯飞ICDAR 2023:图文识别再创辉煌,收获四项冠军

iFlytek's image and text recognition technology continues to make breakthroughs in algorithms, from single word recognition to text line recognition. to more complex two-dimensional structure recognition and chapter-level recognition. More powerful image and text recognition technology can improve the effect and potential of multi-modal large models in image description, image question and answer, image recognition creation, document understanding and processing, etc.

At the same time, image and text recognition technology is also Combining speech recognition, speech synthesis, machine translation and other technologies to form a systematic innovation, the empowered product shows more powerful functions and more obvious value advantages after application. The related project also won the first prize of the 2022 Wu Wenjun Artificial Intelligence Technology Progress Award . In the new journey, "more blooms" in several ICDAR 2023 competitions are not only feedback from iFlytek's continuous progress in the depth of image and text recognition and understanding technology, but also affirmation of its continuous expansion in breadth.

The above is the detailed content of iFLYTEK ICDAR 2023: Image and text recognition achieves greater glory again, winning four championships. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:机器之心. If there is any infringement, please contact admin@php.cn delete
The AI Skills Gap Is Slowing Down Supply ChainsThe AI Skills Gap Is Slowing Down Supply ChainsApr 26, 2025 am 11:13 AM

The term "AI-ready workforce" is frequently used, but what does it truly mean in the supply chain industry? According to Abe Eshkenazi, CEO of the Association for Supply Chain Management (ASCM), it signifies professionals capable of critic

How One Company Is Quietly Working To Transform AI ForeverHow One Company Is Quietly Working To Transform AI ForeverApr 26, 2025 am 11:12 AM

The decentralized AI revolution is quietly gaining momentum. This Friday in Austin, Texas, the Bittensor Endgame Summit marks a pivotal moment, transitioning decentralized AI (DeAI) from theory to practical application. Unlike the glitzy commercial

Nvidia Releases NeMo Microservices To Streamline AI Agent DevelopmentNvidia Releases NeMo Microservices To Streamline AI Agent DevelopmentApr 26, 2025 am 11:11 AM

Enterprise AI faces data integration challenges The application of enterprise AI faces a major challenge: building systems that can maintain accuracy and practicality by continuously learning business data. NeMo microservices solve this problem by creating what Nvidia describes as "data flywheel", allowing AI systems to remain relevant through continuous exposure to enterprise information and user interaction. This newly launched toolkit contains five key microservices: NeMo Customizer handles fine-tuning of large language models with higher training throughput. NeMo Evaluator provides simplified evaluation of AI models for custom benchmarks. NeMo Guardrails implements security controls to maintain compliance and appropriateness

AI Paints A New Picture For The Future Of Art And DesignAI Paints A New Picture For The Future Of Art And DesignApr 26, 2025 am 11:10 AM

AI: The Future of Art and Design Artificial intelligence (AI) is changing the field of art and design in unprecedented ways, and its impact is no longer limited to amateurs, but more profoundly affecting professionals. Artwork and design schemes generated by AI are rapidly replacing traditional material images and designers in many transactional design activities such as advertising, social media image generation and web design. However, professional artists and designers also find the practical value of AI. They use AI as an auxiliary tool to explore new aesthetic possibilities, blend different styles, and create novel visual effects. AI helps artists and designers automate repetitive tasks, propose different design elements and provide creative input. AI supports style transfer, which is to apply a style of image

How Zoom Is Revolutionizing Work With Agentic AI: From Meetings To MilestonesHow Zoom Is Revolutionizing Work With Agentic AI: From Meetings To MilestonesApr 26, 2025 am 11:09 AM

Zoom, initially known for its video conferencing platform, is leading a workplace revolution with its innovative use of agentic AI. A recent conversation with Zoom's CTO, XD Huang, revealed the company's ambitious vision. Defining Agentic AI Huang d

The Existential Threat To UniversitiesThe Existential Threat To UniversitiesApr 26, 2025 am 11:08 AM

Will AI revolutionize education? This question is prompting serious reflection among educators and stakeholders. The integration of AI into education presents both opportunities and challenges. As Matthew Lynch of The Tech Edvocate notes, universit

The Prototype: American Scientists Are Looking For Jobs AbroadThe Prototype: American Scientists Are Looking For Jobs AbroadApr 26, 2025 am 11:07 AM

The development of scientific research and technology in the United States may face challenges, perhaps due to budget cuts. According to Nature, the number of American scientists applying for overseas jobs increased by 32% from January to March 2025 compared with the same period in 2024. A previous poll showed that 75% of the researchers surveyed were considering searching for jobs in Europe and Canada. Hundreds of NIH and NSF grants have been terminated in the past few months, with NIH’s new grants down by about $2.3 billion this year, a drop of nearly one-third. The leaked budget proposal shows that the Trump administration is considering sharply cutting budgets for scientific institutions, with a possible reduction of up to 50%. The turmoil in the field of basic research has also affected one of the major advantages of the United States: attracting overseas talents. 35

All About Open AI's Latest GPT 4.1 Family - Analytics VidhyaAll About Open AI's Latest GPT 4.1 Family - Analytics VidhyaApr 26, 2025 am 10:19 AM

OpenAI unveils the powerful GPT-4.1 series: a family of three advanced language models designed for real-world applications. This significant leap forward offers faster response times, enhanced comprehension, and drastically reduced costs compared t

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

MinGW - Minimalist GNU for Windows

MinGW - Minimalist GNU for Windows

This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

PhpStorm Mac version

PhpStorm Mac version

The latest (2018.2.1) professional PHP integrated development tool

SublimeText3 Linux new version

SublimeText3 Linux new version

SublimeText3 Linux latest version

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

Dreamweaver Mac version

Dreamweaver Mac version

Visual web development tools