


A complete set of tutorials for adapting the Diffusers framework is here! From T2I-Adapter to the popular ControlNet
Not long after ChatGPT came out of the circle, the emergence of ControlNet quickly gained many developers and ordinary users on the English and Chinese Internet. Some users even promoted that the emergence of ControlNet brought AI creation into the era of upright walking. . It is no exaggeration to say that, including ControlNet, the T2I-Adapter, Composer, and LoRA training techniques of the same period, controllable generation, as the last high wall of AI creation, is very likely to have further breakthroughs in the foreseeable time, thus greatly Reduce the user's creation costs and improve the playability of creation. In just two weeks since ControlNet was open sourced, its official Star count has exceeded 10,000. This popularity is undoubtedly unprecedented.
At the same time, the open source community has also greatly lowered the threshold for users. For example, the Hugging Face platform provides basic model weights and general model training framework diffusers, stable-diffusion-webui A complete Demo platform has been developed, and Civitai has contributed a large number of stylized LoRA weights.
##Although webui is currently the most popular The visualization tool has quickly supported various recently launched generative models and supports many options for users to set. Because it focuses on the ease of use of the front-end interface, the code structure behind it is actually very complex and not friendly enough for developers. For example, although webui supports multiple types of loading and inference, it cannot support conversion under different frameworks, nor can it support flexible training of models. In community discussions, we discovered many pain points that have not yet been solved by existing open source code.
First of all, the code framework is not compatible. Currently popular models, such as ControlNet and T2I-Adapter, are not compatible with the mainstream Stable Diffusion training library. diffusers is not compatible, ControlNet pre-trained models cannot be used directly in the diffusers framework.
Secondly, Model loading is limited. Currently, models are saved in various formats, such as .bin, .ckpt, .pth, .satetensors, etc. , in addition to webui, the diffusers framework currently has limited support for these model formats. Considering that most LoRA models are mainly saved in safetensors, it is difficult for users to directly load LoRA models into existing models trained based on the diffusers framework.
Third, The basic model is limited. Currently, ControlNet and T2I-Adapter are trained based on Stable-Diffusion-1.5, and only The model weights under SD1.5 are open sourced. Considering specific scenarios, high-quality animation models such as anything-v4 and ChilloutMix already exist. Even if controllable information is introduced, the final generated results are still limited by the capabilities of UNet in SD1.5.
Finally, model training is limited. Currently, LoRA has been widely verified to be one of the most effective methods for style transfer and maintaining a specific image IP. 1. However, the diffusers framework currently only supports UNet's LoRA embedding and cannot support text encoder embedding, which will limit LoRA training.
After discussing with the open source community, we learned that the diffusers framework, as a general code library, is planning to adapt to the recently launched generation models at the same time; because it involves rewriting many underlying interfaces, it is still It will take some time to update. To this end, we started from the above actual problems and took the lead in proposing self-developed solutions for each problem to quickly help developers develop more easily.
Full adaptation solution from LoRA, ControlNet, T2I-Adapter to diffusers##LoRA for diffusers
This solution is to flexibly embed LoRA weights in various formats in the diffusers framework, that is, the model saved based on diffusers training. Since the training of LoRA usually freezes the base model, it can be easily embedded into existing models as pluggable modules as style or IP conditional constraints. LoRA itself is a general training technique. Its basic principle is that through low-rank decomposition, the number of parameters of the module can be greatly reduced. Currently, in image generation, it is generally used to train pluggable modules that are independent of the base model. , the actual use is to merge it with the output of the base model in the form of residuals.The first is the embedding of LoRA weights. Currently, the weights provided on the Civitai platform are mainly stored in ckpt or safetensors format, divided into the following two situations. (1) Full model (base model LoRA module) If the full model is in safetensors format, it can be converted by the following diffusers script If the full model is in ckpt format, it can be converted through the following diffusers script After the conversion is completed, the model can be loaded directly using the API of diffusers (2) LoRA only (only contains LoRA module) Currently diffusers officially cannot support loading only LoRA weights, and on the open source platform The LoRA weights are basically stored in this form. Essentially, it completes the remapping of key-value in LoRA weights and adapts it to the diffusers model. For this reason, we support this feature ourselves and provide conversion scripts. Only need to specify the model in diffusers format, and the LoRA weights stored in safetensors format. We provide an example conversion. In addition, due to its lightweight, LoRA itself can quickly complete training with small data and can be embedded into other networks. In order not to be limited to the existing LoRA weights, we support multi-module (UNet text encoder) training of LoRA in the diffusers framework and have submitted a PR in the official code base (https://github.com/huggingface/diffusers/pull/ 2479), and supports training LoRA in ColossalAI. The code is open source at: https://github.com/haofanwang/Lora-for-Diffusers ControlNet for diffuserspython ./scripts/convert_original_stable_diffusion_to_diffusers.py --checkpoint_path xxx.safetensors--dump_path save_dir --from_safetensors
python ./scripts/convert_original_stable_diffusion_to_diffusers.py --checkpoint_path xxx.ckpt--dump_path save_dir
from diffusers import StableDiffusionPipeline
pipeline = StableDiffusionPipeline.from_pretrained (save_dir,torch_dtype=torch.float32)
pipeline = StableDiffusionPipeline.from_pretrained (model_id,torch_dtype=torch.float32)
model_path = "onePieceWanoSagaStyle_v2Offset.safetensors"
state_dict = load_file (model_path)
# the default mergering ratio is 0.75, you can manually set it
python convert_lora_safetensor_to_diffusers.py
##This solution is to support the use of ControlNet in the diffusers framework. Based on some attempts of the open source community, we provide a complete use case for ControlNet Anything-V3, supporting the replacement of the base model from the original SD1.5 to the anything-v3 model, so that ControlNet has better animation generation capabilities.
In addition, we also support ControlNet Inpainting and provide a pipeline adapted to diffusers,
and Multi-ControlNet for multi-condition control.
The code is open source at: https://github.com/haofanwang/ControlNet-for-Diffusers
T2I-Adapter for diffusers
The code is open source at: https://github.com/haofanwang/T2I-Adapter-for-Diffusers Currently, the above three adaptation solutions have been open sourced to the community, and have been officially acknowledged in ControlNet and T2I-Adapter respectively. They have also received thanks from the author of stable-diffusion-webui-colab. We are maintaining discussions with diffusers officials and will complete the integration of the above solution into the official code base in the near future. You are also welcome to try our work in advance. If you have any questions, you can directly raise an issue and we will reply as soon as possible.
The above is the detailed content of A complete set of tutorials for adapting the Diffusers framework is here! From T2I-Adapter to the popular ControlNet. For more information, please follow other related articles on the PHP Chinese website!

Large language models (LLMs) have surged in popularity, with the tool-calling feature dramatically expanding their capabilities beyond simple text generation. Now, LLMs can handle complex automation tasks such as dynamic UI creation and autonomous a

Can a video game ease anxiety, build focus, or support a child with ADHD? As healthcare challenges surge globally — especially among youth — innovators are turning to an unlikely tool: video games. Now one of the world’s largest entertainment indus

“History has shown that while technological progress drives economic growth, it does not on its own ensure equitable income distribution or promote inclusive human development,” writes Rebeca Grynspan, Secretary-General of UNCTAD, in the preamble.

Easy-peasy, use generative AI as your negotiation tutor and sparring partner. Let’s talk about it. This analysis of an innovative AI breakthrough is part of my ongoing Forbes column coverage on the latest in AI, including identifying and explaining

The TED2025 Conference, held in Vancouver, wrapped its 36th edition yesterday, April 11. It featured 80 speakers from more than 60 countries, including Sam Altman, Eric Schmidt, and Palmer Luckey. TED’s theme, “humanity reimagined,” was tailor made

Joseph Stiglitz is renowned economist and recipient of the Nobel Prize in Economics in 2001. Stiglitz posits that AI can worsen existing inequalities and consolidated power in the hands of a few dominant corporations, ultimately undermining economic

Graph Databases: Revolutionizing Data Management Through Relationships As data expands and its characteristics evolve across various fields, graph databases are emerging as transformative solutions for managing interconnected data. Unlike traditional

Large Language Model (LLM) Routing: Optimizing Performance Through Intelligent Task Distribution The rapidly evolving landscape of LLMs presents a diverse range of models, each with unique strengths and weaknesses. Some excel at creative content gen


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Dreamweaver Mac version
Visual web development tools

EditPlus Chinese cracked version
Small size, syntax highlighting, does not support code prompt function

SublimeText3 Linux new version
SublimeText3 Linux latest version

mPDF
mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

MinGW - Minimalist GNU for Windows
This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.