ECCV 2024｜Did you really see it, or did you think you saw it? The over-reliance of large multi-modal models on text pre-training knowledge should be resolved

The AIxiv column is a column where this site publishes academic and technical content. In the past few years, the AIxiv column of this site has received more than 2,000 reports, covering top laboratories from major universities and companies around the world, effectively promoting academic exchanges and dissemination. If you have excellent work that you want to share, please feel free to contribute or contact us for reporting. Submission email: liyazhou@jiqizhixin.com; zhaoyunfeng@jiqizhixin.com

Pi Renjie, the first author of this article, is a third-year doctoral student at Hong Kong University of Science and Technology, studying under Professor Zhang Tong and Professor Zhou Xiaofang. Previously received a bachelor's degree in computer engineering from the University of Hong Kong. His research interests include multimodal large language models, data-centric artificial intelligence, and automated machine learning.

With the advancement of large language models (LLMs), multimodal large language models (MLLMs) are developing rapidly. They use pre-trained visual encoders to process images, and input images to LLMs as token embeddings along with text information, thus extending the model's conversational capabilities for processing image inputs. This improvement in capabilities brings possibilities for a variety of potential application areas such as autonomous driving and medical assistants.

Although MLLMs have excellent image and text understanding capabilities, they still suffer from errors or hallucinations, generating responses that do not match the input image, such as answering non-existent objects or misidentifying attributes. We believe that the imbalance of data volume and training time in different training stages of multi-modal large models is one of the main reasons for this type of bias. The language modules of large multi-modal models often use massive text data for pre-training, while the modal alignment stage uses smaller data size and shorter training time.

In order to solve the above problems, we propose a preference alignment method - Bootstrapped Preference Optimization (BPO), which can alleviate the hallucination phenomenon of multi-modal large models while improving the visual understanding ability of the model.

• Paper title: Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization
• Paper link: https://arxiv.org/pdf/2403.08730
• Code link: https://github. com/pipilurj/bootstrapped-preference-optimization-BPO-

Specifically, we designed two methods to automatically construct negative samples for preference learning, exposing the over-reliance of multi-modal models on training. Afterwards, we use the original data annotations as positive samples to fine-tune the preferences of the multi-modal model. Overall, our main contributions are:

1. We propose a new perspective that transforms the multi-modal alignment problem into a preference learning task, where pre-training bias and visual understanding ability are treated as old and new preferences;

2. We introduce a method to automate the construction of large-scale preference datasets. Through this method, a large number of negative samples with pre-training bias information can be constructed;

3. A large number of experiments have proven that our method can effectively improve the cognitive ability of multi-modal large models for images, training The latter model has improved performance in multiple benchmarks.

Scalable preference dataset construction

For positive examples of preference datasets, there are already many ready-made datasets designed for supervised fine-tuning, such as high-quality annotated question answering generated by LlaVA and MiniGPT4 Data,ShareGPTV leverages the powerful GPT4-V as a tool to,generate high-quality captions for images. We use these annotated public data sets as positive responses in the preference data set to avoid expensive manual annotation while ensuring high-quality data pairs.

In order to collect negative response data that reflects pre-training bias, we propose two methods.

a. Weaken image prompts: We add noise to the image data in the preference data set to destroy the image features and make the multi-modal large model more inclined to the original pre-trained distribution when answering. The resulting Error responses will contain the inherent bias of the LLM module. As can be seen from the figure, by adding different levels of noise to the image, the probability of the correct answer appearing is smaller, and the probability of the answer with pre-training bias appearing is greater.

b. Error injection: We require the large language model corresponding to the multi-modal large model to directly rewrite the response, and require the model to generate an incorrect answer that is similar but not exactly the same as the answer.

Next, we use direct preference optimization (DPO) to optimize the multi-modal model:

Experimental evaluation

We use the LLaVA model (LLaVA-7B) fine-tuned by BPO -BPO and LLaVA-13B-BPO) tested on MM-Vet, LLaVA-Wild and Object HalBench. MM-Vet and LlaVA-Bench are lists specifically used to measure the comprehensive capabilities of models, while Object HalBench is used to evaluate the visual credibility of multi-modal large models.

Experimental results show that the model fine-tuned by BPO takes the lead in all tasks on the three benchmark lists. On most tasks, LLaVA-7B-BPO even outperforms the untuned LLaVa1.5-13B model.

We also compare BPO with supervised fine-tuning training (SFT). We fine-tune the model by directly using positive samples from the dataset as supervised data. Experiments show that multi-modal large models fine-tuned by BPO perform better than SFT fine-tuning on different categories of subtasks.

In terms of qualitative results, we compared the performance of multi-modal large models before and after BPO fine-tuning. We found that the BPO-finetuned model produced answers that were more faithful to the image input and contained less erroneous information.

For more research details, please refer to the original paper.

The above is the detailed content of ECCV 2024｜Did you really see it, or did you think you saw it? The over-reliance of large multi-modal models on text pre-training knowledge should be resolved. For more information, please follow other related articles on the PHP Chinese website!