7B model surpasses GPT4-V! Hong Kong University of Science and Technology and others released the 'Graph Reasoning Question and Answer' data set GITQA: Visual graphs can improve reasoning capabilities

Graph neural networks (GNNs) are good at leveraging the structural information of graphs for inference, but often require domain-specific tuning to achieve optimal performance, which makes their ability to generalize across different tasks limited. limit.

Large language models (LLMs) have stronger cross-task and generalization capabilities for graph reasoning, but often do not perform as well as dedicated graph neural network models on specific tasks.

Current research on graph reasoning often ignores the importance of visual information in graph reasoning, whether it is traditional graph neural networks or graph reasoning methods based on large language models.

However, humans use visual features to efficiently and accurately complete graph tasks, such as determining whether there are rings in the graph.

Therefore, it is of great significance to explore the role of visual form graph information in graph reasoning.

More specifically, can drawing a graph (Graph) as a picture (Image) give the model special reasoning capabilities? Can these images (called Visual Graphs) enhance existing graph reasoning models based on other modalities?

In order to answer these questions, the research team from Hong Kong University of Science and Technology and Southern University of Science and Technology built the first inference question and answer data set containing visual graphs, GITQA, and used open source models such as GPT-4 turbo, GPT-4V and Extensive experiments have been conducted on closed-source models such as Vicuna and LLaVA, confirming the role of Visual Graph in graph reasoning and its mutual enhancement with text modality.

Picture

Paper address: https://arxiv.org/abs/2402.02130

Project homepage: https://v-graph.github.io/

In the GITQA test benchmark, fine-tuned based on LLaVA-7B/13B The multi-modal model GITA-7B/13B demonstrates graph reasoning performance that surpasses GPT-4V.

GITQA Multimodal Graph Reasoning Question and Answer Dataset

The research team established the GITQA data set and its corresponding Test benchmark, GITQA dataset contains more than 423K question and answer instances, each instance contains corresponding graph structure-text-visual information and its corresponding question and answer pairs.

The GITQA data set contains two versions: GITQA-Base and GITQA-Aug, of which GITQA-Base only contains visual images of a single style.

GITQA-Aug is even richer. It performs a variety of data enhancements on the visual map, including changing the layout, point shape, edge width, point style, etc., thereby providing Provides more diverse visual representations.

Picture

As shown in Figure 1, the GITQA test benchmark contains 8 representative graph reasoning tasks: Connectivity (judgment graph (whether two points in the graph are connected), Cycle (to determine whether there is a cycle in the graph), TS (to find the topological order of the graph), SP (to find the shortest path between two points in the graph), MaxFlow (to calculate the maximum flow between two points in the graph) ), BGM (compute the maximum matching of a bipartite graph), HP (find the Hamiltonian path in the graph) and GNN (simulate the message passing of GNN).

Picture

The data set corresponding to each task is divided into different difficulty levels according to the complexity of the graph structure. A subset of (relevant statistics are shown in Table 1).

Experiments and results

Experiment 1: Comparison of graph reasoning capabilities of models based on different modal graph information

Research The team evaluated popular closed-source methods based on different modal graph input types (including text only (T-Only), vision only (V-Only), and text plus vision (V T)) on the GITQA-Base dataset. and the performance of open source large language models (such as GPT-4 turbo and Vicuna-7B/13B) and large multi-modal language models (such as GPT-4V and LLaVA-7B/13B). as shown in picture 2.

Picture

Specifically, the closed-source models GPT-4 and GPT-4V perform zero-shot inference, while for open-source Models Vicuna and LLaVA were fine-tuned by keeping the backbone model parameters unchanged, and only the training Projector and LoRA parts were fine-tuned (in particular, the LLaVA model after visual text dual-modal fine-tuning was named GITA by the researcher).

Table 2 summarizes the test results for all eight graph reasoning tasks.

Picture

Visual mode V.S. Text mode

As can be seen from Table 2, in Cycle and BGM On tasks, the visual modality performed better than the text modality, while on the other five tasks it was inferior to the text modality. This reveals that vision and text each have advantages in handling specific types of graph reasoning tasks. Mutual enhancement of visual and text modalities

For the closed-source model, GPT-4V (V T) has much higher average accuracy on eight tasks than GPT-4 Turbo (T-only) and GPT-4V (V-only).

For open source models (7B, 13B), similarly, the GITA model trained using bimodal data performed best on average. These observations verify that using visual and textual information simultaneously can enhance the model’s graph reasoning capabilities and achieve better performance than single-modal models.

More specifically, GITA-7B (V T) outperforms LLaVA-7B (V-only) and Vicuna-7B (T-only) in almost all tasks. For the closed-source model, using bimodality achieved the highest accuracy on five out of eight tasks. The fine-tuned LLaVA model can surpass GPT-4V

As shown in Table 2 and Figure 3, the GITA-7B and GITA-13B models, that is, the LLaVA-7B/13B model after dual-modal fine-tuning, show A significant performance improvement of more than 13% compared to GPT-4V. This huge improvement shows that the fine-tuned GITA model can effectively learn excellent graph reasoning capabilities from the GITQA dataset.

Picture

Experiment 2: The impact of difficulty level on graph tasks

Table 3 further shows the performance of the model at different difficulty levels test accuracy, the GNN task was omitted as it was too challenging for all models).

Performance using the visual modality alone outperformed the text modality and was comparable to using both modalities in Cycle and BGM tasks at all difficulty levels.

However, for other tasks, the performance of models using only the visual modality drops significantly when the difficulty increases from easy to medium or hard.

Picture

Similarly, when the difficulty increases, the models using only text modality and using visual text modality also perform better on these tasks. There will be a significant performance drop.

For the Connectivity task, GITA-7B (visual text) and GITA-13B (visual text) show comparable performance at all three challenge levels.

However, this consistent pattern is not observed in GPT-4V (visual text), as its performance decreases with increasing difficulty levels.

Experiment 3: Visual graph enhancement strategies and style preferences

The research team also explored special data enhancement The effectiveness of the strategy in fine-tuning the model.

Based on different enhancement strategies, the researchers divided the GITQA-Aug data set into four enhancement subsets: layout enhancement data set, node shape enhancement data set, and edge width enhancement data Set,node style augmented dataset.

Picture

The researchers performed all four enhancement subsets on the LLaVA-7B model using only visual map information. After separate fine-tuning, the comparison of its inference performance with that before data augmentation is shown in Table 4.

It can be clearly seen that the model's reasoning ability for challenging tasks on the layout-enhanced data set has improved dramatically (SP increased by 64.8%, HP increased by 69.63%).

The other three data enhancement strategies actually lead to performance degradation.

Specifically, the model achieves excellent results on the layout enhancement set, which is more than 11% higher than the GITQA-Base set. In comparison, the average results for the eight tasks in the other augmented sets are about 5% lower than the base set

Picture

These findings suggest that layout-based data augmentation provides a more effective visual perspective for graph reasoning. Furthermore, the researchers also tested the performance of Visual Graph reasoning based on each style within the same group under each enhancement strategy. As shown in Table 5, it shows that the model has no obvious style preference.

The above is the detailed content of 7B model surpasses GPT4-V! Hong Kong University of Science and Technology and others released the 'Graph Reasoning Question and Answer' data set GITQA: Visual graphs can improve reasoning capabilities. For more information, please follow other related articles on the PHP Chinese website!