search
HomeTechnology peripheralsAIThe diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

Recently, many unexpected breakthroughs have been made in the field of text-generated images, and many models can achieve the function of creating high-quality and diverse images based on text instructions. While the images generated are already very realistic, current models are often good at generating images of physical objects such as landscapes and objects, but struggle to generate images with a high degree of coherent detail, such as images with complex glyph text such as Chinese characters.

In order to solve this problem, researchers from OPPO and other institutions have proposed a general learning framework GlyphDraw, which is designed to enable the model to generate images embedded with coherent text. This is the field of image synthesis. The first work to solve the problem of Chinese character generation.

The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

  • ##Paper address: https://arxiv.org/abs/2303.17870
  • Project homepage: https://1073521013.github.io/glyph-draw.github.io/

Let’s start with Let’s take a look at the generation effect, for example, generating warning slogans for the exhibition hall:

The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

Generating billboards:

The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

Add a brief text description to the picture. The text style can also be diverse:

The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

Also, the most interesting and practical example is to generate emoticons:

The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

Although the result has some flaws , but the overall generation effect is already very good. Overall, the main contributions of this research include:


  • This research proposes the first Chinese character image generation framework GlyphDraw, which utilizes some Auxiliary information, including Chinese character glyphs and positions, provides fine-grained guidance throughout the generation process, allowing Chinese character images to be seamlessly embedded into images with high quality;
  • This study proposes an effective The training strategy limits the number of trainable parameters in the pre-trained model to prevent overfitting and catastrophic forgetting, effectively maintaining the model's powerful open domain generation performance while achieving accurate Chinese character image generation. .
  • This study introduces the construction process of the training dataset and proposes a new benchmark to evaluate the quality of Chinese character image generation using OCR models. Among them, GlyphDraw achieved a generation accuracy of 75%, significantly better than previous image synthesis methods.

The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

Model Introduction

The study first designed a complex image-text data set Build a strategy, and then propose a general learning framework GlyphDraw based on the open source image synthesis algorithm Stable Diffusion, as shown in Figure 2 below.

The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

The overall training goal of Stable Diffusion can be expressed as the following formula:

The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

GlyphDraw is based on the cross-attention mechanism in Stable Diffusion, where the original input latent vector z_t is replaced by a concatenation of the image latent vector z_t, the text mask l_m, and the glyph image l_g.

The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

Furthermore, Condition C is equipped with hybrid glyph and text features by using domain-specific fusion modules. The introduction of text mask and glyph information allows the entire training process to achieve fine-grained diffusion control, which is a key component to improving model performance, and ultimately generates images with Chinese character text.

Specifically, the pixel representation of text information, especially complex text forms such as pictographic Chinese characters, is significantly different from natural objects. For example, the Chinese word "sky" is composed of multiple strokes in a two-dimensional structure, and its corresponding natural image is "blue sky dotted with white clouds." In contrast, Chinese characters have very fine-grained characteristics, and even small movements or deformations can lead to incorrect text rendering, making image generation impossible.

Embedding characters in natural image backgrounds also requires consideration of a key issue, which is to accurately control the generation of text pixels while avoiding affecting adjacent natural image pixels. In order to render perfect Chinese characters on natural images, the authors carefully designed two key components integrated into the diffusion synthesis model, namely position control and glyph control.

Different from the global conditional input of other models, character generation requires more attention to specific local areas of the image because the latent feature distribution of character pixels is different from that of natural image pixels. Huge difference. In order to prevent model learning from collapsing, this study innovatively proposes fine-grained location area control to decouple the distribution between different areas.

In addition to position control, another important issue is the fine control of Chinese character stroke synthesis. Considering the complexity and diversity of Chinese characters, it is extremely difficult to simply learn from large image-text datasets without any explicit prior knowledge. In order to accurately generate Chinese characters, this study incorporates explicit glyph images as additional conditional information into the model diffusion process.

The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

Experiments and results

Since there is no data set specifically used for Chinese character image generation, this study first constructed a The benchmark data set ChineseDrawText was used for qualitative and quantitative evaluation, and then the generation accuracy of several methods (evaluated by the OCR recognition model) was tested and compared on ChineseDrawText.

The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

#The GlyphDraw model proposed in this study demonstrates that it achieves an average accuracy of 75% by effectively using auxiliary glyph and position information. Excellent character image generation capabilities. The visual comparison results of several methods are shown in the figure below:

The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

In addition, GlyphDraw can also maintain open domain image synthesis performance by limiting training parameters, The FID of general image synthesis only dropped by 2.3 on MS-COCO FID-10k.

The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw

Interested readers can read the original text of the paper to learn more about the research details .

The above is the detailed content of The diffusion model generates images with Chinese characters and outputs emoticons with one click: OPPO and others proposed GlyphDraw. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:51CTO.COM. If there is any infringement, please contact admin@php.cn delete
Tesla's Robovan Was The Hidden Gem In 2024's Robotaxi TeaserTesla's Robovan Was The Hidden Gem In 2024's Robotaxi TeaserApr 22, 2025 am 11:48 AM

Since 2008, I've championed the shared-ride van—initially dubbed the "robotjitney," later the "vansit"—as the future of urban transportation. I foresee these vehicles as the 21st century's next-generation transit solution, surpas

Sam's Club Bets On AI To Eliminate Receipt Checks And Enhance RetailSam's Club Bets On AI To Eliminate Receipt Checks And Enhance RetailApr 22, 2025 am 11:29 AM

Revolutionizing the Checkout Experience Sam's Club's innovative "Just Go" system builds on its existing AI-powered "Scan & Go" technology, allowing members to scan purchases via the Sam's Club app during their shopping trip.

Nvidia's AI Omniverse Expands At GTC 2025Nvidia's AI Omniverse Expands At GTC 2025Apr 22, 2025 am 11:28 AM

Nvidia's Enhanced Predictability and New Product Lineup at GTC 2025 Nvidia, a key player in AI infrastructure, is focusing on increased predictability for its clients. This involves consistent product delivery, meeting performance expectations, and

Exploring the Capabilities of Google's Gemma 2 ModelsExploring the Capabilities of Google's Gemma 2 ModelsApr 22, 2025 am 11:26 AM

Google's Gemma 2: A Powerful, Efficient Language Model Google's Gemma family of language models, celebrated for efficiency and performance, has expanded with the arrival of Gemma 2. This latest release comprises two models: a 27-billion parameter ver

The Next Wave of GenAI: Perspectives with Dr. Kirk Borne - Analytics VidhyaThe Next Wave of GenAI: Perspectives with Dr. Kirk Borne - Analytics VidhyaApr 22, 2025 am 11:21 AM

This Leading with Data episode features Dr. Kirk Borne, a leading data scientist, astrophysicist, and TEDx speaker. A renowned expert in big data, AI, and machine learning, Dr. Borne offers invaluable insights into the current state and future traje

AI For Runners And Athletes: We're Making Excellent ProgressAI For Runners And Athletes: We're Making Excellent ProgressApr 22, 2025 am 11:12 AM

There were some very insightful perspectives in this speech—background information about engineering that showed us why artificial intelligence is so good at supporting people’s physical exercise. I will outline a core idea from each contributor’s perspective to demonstrate three design aspects that are an important part of our exploration of the application of artificial intelligence in sports. Edge devices and raw personal data This idea about artificial intelligence actually contains two components—one related to where we place large language models and the other is related to the differences between our human language and the language that our vital signs “express” when measured in real time. Alexander Amini knows a lot about running and tennis, but he still

Jamie Engstrom On Technology, Talent And Transformation At CaterpillarJamie Engstrom On Technology, Talent And Transformation At CaterpillarApr 22, 2025 am 11:10 AM

Caterpillar's Chief Information Officer and Senior Vice President of IT, Jamie Engstrom, leads a global team of over 2,200 IT professionals across 28 countries. With 26 years at Caterpillar, including four and a half years in her current role, Engst

New Google Photos Update Makes Any Photo Pop With Ultra HDR QualityNew Google Photos Update Makes Any Photo Pop With Ultra HDR QualityApr 22, 2025 am 11:09 AM

Google Photos' New Ultra HDR Tool: A Quick Guide Enhance your photos with Google Photos' new Ultra HDR tool, transforming standard images into vibrant, high-dynamic-range masterpieces. Ideal for social media, this tool boosts the impact of any photo,

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Atom editor mac version download

Atom editor mac version download

The most popular open source editor

SublimeText3 Linux new version

SublimeText3 Linux new version

SublimeText3 Linux latest version

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.