


Smaller files, higher quality, can the popular Stable Diffusion compress images?
Recently, Stable Diffusion has become an emerging research direction. A blogger named Matthias Bühlmann tried to experimentally explore the power of this model and found that Stable Diffusion is a very powerful lossy image compression codec. He wrote a blog describing this experimental analysis process. The following is the original blog text.
First of all, Matthias Bühlmann gives the compression results of the Stable Diffusion method and JPG and WebP under high compression factor conditions. All results are at a resolution of 512x512 pixels:
San Francisco landscape, from left to right: JPG (6.16kB), WebP (6.80kB), Stable Diffusion: (4.96kB).
Candy shop, from left to right: JPG (5.68kB), WebP (5.71 kB), Stable Diffusion (4.98kB).
#Animal photos, left to right: JPG (5.66 kB), WebP (6.74kB), Stable Diffusion (4.97kB).
#These examples clearly show that compressing images with Stable Diffusion preserves better image quality at smaller file sizes compared to JPG and WebP.
Exploration ExperimentMatthias Bühlmann analyzed the working principle. Stable Diffusion uses three series-trained artificial neural networks:
- Variational Auto Encoder (VAE)
- U-Net
- Text encoding Text Encoder
VAE encodes and decodes images in image space into some underlying spatial representation. The latent spatial representation of the source image (512 x 512, 3x8 or 4x8 bit) will have a lower resolution (64 x 64) and a higher accuracy (4x32 bit).
VAE learns by itself during the training process. As the model is gradually trained, the latent space representation of different versions of the model may look different, such as the latent space representation of Stable Diffusion v1.4 The spatial representation is as follows (remapped to a 4-channel color image):
When re-expanded and interpreted latent features into color values (using alpha channel ), the main features of the image are still visible, and VAE also encodes higher-resolution features into pixel values.
For example, through a VAE encoding/decoding roundtrip, the following results are obtained:
It is worth noting that this roundtrip is not lossless. For example, the white words on the blue tape in the picture are slightly less readable after decoding. The VAE of the Stable Diffusion v1.4 model is generally not very good at representing small text and faces.
We know that the main purpose of Stable Diffusion is to generate images based on text descriptions, which requires the model to operate on the latent spatial representation of the image. The model uses a trained U-Net to iteratively denoise the latent space image, outputting what it "sees" (predicts) in the noise, similar to how we sometimes see clouds as shapes or faces. In the iterative denoising step, a third ML model (text encoder) guides U-Net to try to see different information.
Matthias Bühlmann analyzes how the latent representation generated by VAE can be effectively compressed. He found that sampling the latent representation in VAE or applying existing lossy image compression methods to the latent representation significantly degrades the quality of the reconstructed image, while the VAE decoding process appears to be relatively robust to the quality of the latent representation.
Matthias Bühlmann quantized the latent representation from floating point to 8-bit unsigned integers and found only very small reconstruction errors. As shown in the figure below, left: 32-bit floating point potential representation; middle: ground truth; right: 8-bit integer potential representation.
He also found that through further quantization through palette and dithering algorithms, the results obtained would be unexpectedly good. However, when decoding directly using VAE, the palettized representation leads to some visible artifacts:
Left: 32-bit latent representation; Middle: 8-bit quantized latent representation; Right: palettized 8-bit latent representation with Floyd-Steinberg dither
Palettized representation with Floyd-Steinberg jitter introduces noise, distorting the decoding results. So Matthias Bühlmann used U-Net to remove the noise caused by jitter. After 4 iterations, the reconstructed result is visually very close to the unquantized version:
Reconstructed result (left : palettized representation with Floyd-Steinberg jitter; middle: denoising after four iterations; right: Ground Truth).
#While the results are very good, some artifacts are introduced, such as the glossy shadow on the center symbol above.
Although subjectively, the results of Stable Diffusion compressed images are much better than JPG and WebP, from the perspective of PSNR, SSIM and other indicators, Stable Diffusion has no obvious advantages.
As shown in the figure below, although Stable Diffusion as a codec is much better than other methods in retaining image granularity, it is affected by compression artifacts, the shape of objects in the image, etc. Characteristics subject to change.
Left: JPG compression; middle: Ground Truth; right: Stable Diffusion compression.
It is worth noting that the current Stable Diffusion v1.4 model cannot well preserve text information and facial features with small fonts during the compression process , but the Stable Diffusion v1.5 model has improved in face generation.
##Left: Ground Truth; Middle: after VAE roundtrip (32-bit latent features) ; Right: Results of decoding from palettized denoised 8-bit latent features.
After the blog was published, Matthias Bühlmann’s experimental analysis aroused everyone’s discussion.
Matthias Bühlmann himself believes that the image compression effect of Stable Diffusion is better than expected, and U-Net seems to be able to effectively eliminate the noise introduced by dithering. However, future versions of the Stable Diffusion model may no longer have this image compression feature.
However, some netizens questioned: "VAE itself is used for image compression." For example, the Transformer-based image compression method TIC uses VAE architecture, so Matthias Bühlmann's experiment seems to be overkill.
What do you think of this?
The above is the detailed content of Smaller files, higher quality, can the popular Stable Diffusion compress images?. For more information, please follow other related articles on the PHP Chinese website!

Exploring the Inner Workings of Language Models with Gemma Scope Understanding the complexities of AI language models is a significant challenge. Google's release of Gemma Scope, a comprehensive toolkit, offers researchers a powerful way to delve in

Unlocking Business Success: A Guide to Becoming a Business Intelligence Analyst Imagine transforming raw data into actionable insights that drive organizational growth. This is the power of a Business Intelligence (BI) Analyst – a crucial role in gu

SQL's ALTER TABLE Statement: Dynamically Adding Columns to Your Database In data management, SQL's adaptability is crucial. Need to adjust your database structure on the fly? The ALTER TABLE statement is your solution. This guide details adding colu

Introduction Imagine a bustling office where two professionals collaborate on a critical project. The business analyst focuses on the company's objectives, identifying areas for improvement, and ensuring strategic alignment with market trends. Simu

Excel data counting and analysis: detailed explanation of COUNT and COUNTA functions Accurate data counting and analysis are critical in Excel, especially when working with large data sets. Excel provides a variety of functions to achieve this, with the COUNT and COUNTA functions being key tools for counting the number of cells under different conditions. Although both functions are used to count cells, their design targets are targeted at different data types. Let's dig into the specific details of COUNT and COUNTA functions, highlight their unique features and differences, and learn how to apply them in data analysis. Overview of key points Understand COUNT and COU

Google Chrome's AI Revolution: A Personalized and Efficient Browsing Experience Artificial Intelligence (AI) is rapidly transforming our daily lives, and Google Chrome is leading the charge in the web browsing arena. This article explores the exciti

Reimagining Impact: The Quadruple Bottom Line For too long, the conversation has been dominated by a narrow view of AI’s impact, primarily focused on the bottom line of profit. However, a more holistic approach recognizes the interconnectedness of bu

Things are moving steadily towards that point. The investment pouring into quantum service providers and startups shows that industry understands its significance. And a growing number of real-world use cases are emerging to demonstrate its value out


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

SecLists
SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

PhpStorm Mac version
The latest (2018.2.1) professional PHP integrated development tool

DVWA
Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

Dreamweaver Mac version
Visual web development tools

Dreamweaver CS6
Visual web development tools