


Recently, many unexpected breakthroughs have been made in the field of text-generated images, and many models can achieve the function of creating high-quality and diverse images based on text instructions. While the images generated are already very realistic, current models are often good at generating images of physical objects such as landscapes and objects, but struggle to generate images with a high degree of coherent detail, such as images with complex glyph text such as Chinese characters.
In order to solve this problem, researchers from OPPO and other institutions have proposed a general learning framework GlyphDraw, which is designed to enable the model to generate images embedded with coherent text. This is the field of image synthesis. The first work to solve the problem of Chinese character generation.
- ##Paper address: https://arxiv.org/abs/2303.17870
- Project homepage: https://1073521013.github.io/glyph-draw.github.io/
Let’s start with Let’s take a look at the generation effect, for example, generating warning slogans for the exhibition hall:
Generating billboards:
Add a brief text description to the picture. The text style can also be diverse:
Also, the most interesting and practical example is to generate emoticons:
Although the result has some flaws , but the overall generation effect is already very good. Overall, the main contributions of this research include:
- This research proposes the first Chinese character image generation framework GlyphDraw, which utilizes some Auxiliary information, including Chinese character glyphs and positions, provides fine-grained guidance throughout the generation process, allowing Chinese character images to be seamlessly embedded into images with high quality;
- This study proposes an effective The training strategy limits the number of trainable parameters in the pre-trained model to prevent overfitting and catastrophic forgetting, effectively maintaining the model's powerful open domain generation performance while achieving accurate Chinese character image generation. .
- This study introduces the construction process of the training dataset and proposes a new benchmark to evaluate the quality of Chinese character image generation using OCR models. Among them, GlyphDraw achieved a generation accuracy of 75%, significantly better than previous image synthesis methods.
The study first designed a complex image-text data set Build a strategy, and then propose a general learning framework GlyphDraw based on the open source image synthesis algorithm Stable Diffusion, as shown in Figure 2 below.
The overall training goal of Stable Diffusion can be expressed as the following formula:
GlyphDraw is based on the cross-attention mechanism in Stable Diffusion, where the original input latent vector z_t is replaced by a concatenation of the image latent vector z_t, the text mask l_m, and the glyph image l_g.
Furthermore, Condition C is equipped with hybrid glyph and text features by using domain-specific fusion modules. The introduction of text mask and glyph information allows the entire training process to achieve fine-grained diffusion control, which is a key component to improving model performance, and ultimately generates images with Chinese character text.
Specifically, the pixel representation of text information, especially complex text forms such as pictographic Chinese characters, is significantly different from natural objects. For example, the Chinese word "sky" is composed of multiple strokes in a two-dimensional structure, and its corresponding natural image is "blue sky dotted with white clouds." In contrast, Chinese characters have very fine-grained characteristics, and even small movements or deformations can lead to incorrect text rendering, making image generation impossible.
Embedding characters in natural image backgrounds also requires consideration of a key issue, which is to accurately control the generation of text pixels while avoiding affecting adjacent natural image pixels. In order to render perfect Chinese characters on natural images, the authors carefully designed two key components integrated into the diffusion synthesis model, namely position control and glyph control.
Different from the global conditional input of other models, character generation requires more attention to specific local areas of the image because the latent feature distribution of character pixels is different from that of natural image pixels. Huge difference. In order to prevent model learning from collapsing, this study innovatively proposes fine-grained location area control to decouple the distribution between different areas.
In addition to position control, another important issue is the fine control of Chinese character stroke synthesis. Considering the complexity and diversity of Chinese characters, it is extremely difficult to simply learn from large image-text datasets without any explicit prior knowledge. In order to accurately generate Chinese characters, this study incorporates explicit glyph images as additional conditional information into the model diffusion process.
Experiments and results
Since there is no data set specifically used for Chinese character image generation, this study first constructed a The benchmark data set ChineseDrawText was used for qualitative and quantitative evaluation, and then the generation accuracy of several methods (evaluated by the OCR recognition model) was tested and compared on ChineseDrawText.
#The GlyphDraw model proposed in this study demonstrates that it achieves an average accuracy of 75% by effectively using auxiliary glyph and position information. Excellent character image generation capabilities. The visual comparison results of several methods are shown in the figure below:
In addition, GlyphDraw can also maintain open domain image synthesis performance by limiting training parameters, The FID of general image synthesis only dropped by 2.3 on MS-COCO FID-10k.
Interested readers can read the original text of the paper to learn more about the research details .
The above is the detailed content of The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw. For more information, please follow other related articles on the PHP Chinese website!

Since 2008, I've championed the shared-ride van—initially dubbed the "robotjitney," later the "vansit"—as the future of urban transportation. I foresee these vehicles as the 21st century's next-generation transit solution, surpas

Revolutionizing the Checkout Experience Sam's Club's innovative "Just Go" system builds on its existing AI-powered "Scan & Go" technology, allowing members to scan purchases via the Sam's Club app during their shopping trip.

Nvidia's Enhanced Predictability and New Product Lineup at GTC 2025 Nvidia, a key player in AI infrastructure, is focusing on increased predictability for its clients. This involves consistent product delivery, meeting performance expectations, and

Google's Gemma 2: A Powerful, Efficient Language Model Google's Gemma family of language models, celebrated for efficiency and performance, has expanded with the arrival of Gemma 2. This latest release comprises two models: a 27-billion parameter ver

This Leading with Data episode features Dr. Kirk Borne, a leading data scientist, astrophysicist, and TEDx speaker. A renowned expert in big data, AI, and machine learning, Dr. Borne offers invaluable insights into the current state and future traje

There were some very insightful perspectives in this speech—background information about engineering that showed us why artificial intelligence is so good at supporting people’s physical exercise. I will outline a core idea from each contributor’s perspective to demonstrate three design aspects that are an important part of our exploration of the application of artificial intelligence in sports. Edge devices and raw personal data This idea about artificial intelligence actually contains two components—one related to where we place large language models and the other is related to the differences between our human language and the language that our vital signs “express” when measured in real time. Alexander Amini knows a lot about running and tennis, but he still

Caterpillar's Chief Information Officer and Senior Vice President of IT, Jamie Engstrom, leads a global team of over 2,200 IT professionals across 28 countries. With 26 years at Caterpillar, including four and a half years in her current role, Engst

Google Photos' New Ultra HDR Tool: A Quick Guide Enhance your photos with Google Photos' new Ultra HDR tool, transforming standard images into vibrant, high-dynamic-range masterpieces. Ideal for social media, this tool boosts the impact of any photo,


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Atom editor mac version download
The most popular open source editor

SublimeText3 Linux new version
SublimeText3 Linux latest version

mPDF
mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

Zend Studio 13.0.1
Powerful PHP integrated development environment

SecLists
SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.