HKU Byte proposes a new paradigm of multi-modal large models, simulating human perception first and then cognition, to accurately locate objects in the picture

Currently, Multimodal Large Model (MLLM)has demonstrated strong cognitive understanding capabilities on multiple visual tasks.

However, most large multi-modal models are limited to one-way image understanding, making it difficult to map the understood content back to the image.

For example, the model can easily tell what objects are in the picture, but it cannot accurately identify the objects in the picture.

The lack of positioning capabilities directly limits the application of multi-modal large models in downstream fields such as image editing, autonomous driving, and robot control.

In response to this problem, researchers from the University of Hong Kong and ByteDance’s commercialization team proposed a new paradigm Groma——

Through regional images Encoding to improve the perceptual positioning capabilities of multi-modal large models.

After integrating positioning, Groma can directly connect text content and image areas, thereby significantly improving the interactivity and directionality of conversations. This method does not change the original meaning, but only slightly adjusts the expression.

Core idea

How to give multi-modal large models the ability to locate objects, that is, to associate text content with image areas, Achieving "meaningful words" is a major research hotspot at present. The goal of the multimodal large model is to be able to find the region in the image that corresponds to the description when given an image and a corresponding text description. This task is called the image-text alignment problem. In order to solve this problem,

A common approach is to fine-tune the large language model to directly output object coordinates. However, this method has many limitations:

1. The large language model pre-trained on the text itself does not have the ability to understand space, and it is difficult to accurately locate objects relying only on fine-tuning with a small amount of data.

2. The positioning task has high requirements on the resolution of the input image, but increasing the resolution will significantly increase the calculation amount of the multi-modal large model.

3. The output form of the large language model is not suitable for processing fine positioning tasks, such as segmentation.

Based on these considerations, Groma proposed to transfer the positioning to the vision tokenizer of the multi-modal large model. The vision tokenizer discovers and locates potential objects, and then passes them to the large language model for recognition.

At the same time, this design also makes full use of the spatial understanding ability of the vision tokenizer itself, without the need for external expert models (such as SAM) to assist positioning , thus avoiding the redundancy of external models.

Specifically, Groma introduces region coding to realize the positioning function based on global image coding - as shown in the figure below, Groma first uses Region Proposer to locate potential objects, and then uses Region Encoder to locate potential objects. The regions reached are encoded into region tokens one by one.

The large language model can determine the corresponding region based on the semantic meaning of the region token, and achieve a hyperlink-like effect by inserting the region token into the output to achieve visually grounded conversation.

Similarly, the user-specified region can also be encoded into the corresponding region token through the Region Encoder and inserted into the user command, so that the multi-modal model can focus on the specified region and generate directional answers. .

In order to improve the robustness and accuracy of positioning, Groma uses more than 8M data (including SA1B) to pre-train the Region Proposer. Therefore, the proposals it generates include not only common objects, but also elements such as the components of the objects and the broader background.

In addition, thanks to the separated design, Groma can use high-resolution feature maps for Region Proposer/Encoder input, and use low-resolution feature maps for large model input, thus reducing the cost The calculation amount is reduced without losing positioning performance.

Experimental results

Groma has demonstrated performance surpassing MiniGPT-v2 and Qwen-VL on traditional Grounding Benchmarks.

At the same time, Groma has verified its dialogue and reasoning capabilities on the VQA Benchmark (LLaVA-COCO), which is common to multi-modal large models.

In the visual comparison, Groma also showed higher recall and fewer hallucinations.

In addition, Groma also supports referral dialogue and grounded chat that integrate dialogue capabilities and positioning capabilities.

Thanks to the powerful cognitive reasoning capabilities of large language models, multi-modal large models perform outstandingly in visual understanding tasks.

However, some traditional vision tasks, such as detection segmentation, depth estimation, etc., rely more on visual perception capabilities, which is precisely what large language models lack.

Groma provides a new solution to this problem, which is to decouple perception and cognition, with the vision tokenizer responsible for perception and the large language model responsible for cognition.

This form of perception first and then cognition is not only more in line with the human visual process, but also avoids the computational overhead of retraining a large language model.

On May 15th, ByteDance just announced the self-developed large model of Doubao, which provides multi-modal capabilities, downstream supports 50+ businesses such as Doubao APP, Button, and Jimeng, and is open to the public through the Volcano Engine Enterprise customers, helping enterprises improve efficiency and accelerate intelligent innovation. At present, Doubao APP has become the AIGC application with the largest number of users in the Chinese market. ByteDance is continuing to increase its investment in top talents and cutting-edge technologies, and participate in the industry's top technical challenges and difficulties.

Project website:
https://www.php.cn/link/07a81d45ff030b63fe2a0f375b779f09
Paper link:
##https://www.php.cn/link/b82b80956cfbe75101bd223fe6319dec
Open Source Code:
##https://www.php.cn/link/b984bddf9e7c8fb09854e208c0284764

The above is the detailed content of HKU Byte proposes a new paradigm of multi-modal large models, simulating human perception first and then cognition, to accurately locate objects in the picture. For more information, please follow other related articles on the PHP Chinese website!