The first universal framework in the graph field is here! Selected into ICLR\'24 Spotlight, any data set or classification problem can be solved

Can there be a general graph model——

It can not only predict toxicity based on molecular structure, but also give information about social networks A friend recommended?

Or can it not only predict the citations of papers by different authors, but also discover the human aging mechanism in the gene network?

Don’t tell me, the “One for All(OFA)” framework accepted as Spotlight by ICLR 2024 achieves this "essence".

This research was jointly proposed by researchers such as Professor Chen Yixin’s team at Washington University in St. Louis, Zhang Muhan from Peking University, and Tao Dacheng from JD Research Institute.

As the first general framework in the graph field, OFA enables training a single GNN model to solve the classification task of any data set, any task type, and any scene in the graph field.

How to implement it, the following is the author's contribution.

The design of universal models in the graph field faces three major difficulties

Designing a universal basic model to solve a variety of tasks is a long-term goal in the field of artificial intelligence. In recent years, basic large language models (LLMs) have performed well in processing natural language tasks. However, in the field of graphs, although graph neural networks

(GNNs)

have good performance in different graph data, how to design and train a network that can handle multiple graphs at the same time? The basic graph model of the task still has a long way to go. Compared with the natural language field, the design of general models in the graph field faces many unique difficulties.

First of all, different from natural language, different graph data have completely different attributes and distributions.

For example, a molecular diagram describes how multiple atoms form different chemical substances through different force relationships. The citation relationship diagram describes the network of mutual citations between articles.

These different graph data are difficult to unify under a training framework.

Secondly, unlike all tasks in LLMs, which can be converted into unified context generation tasks, graph tasks include a variety of sub-tasks, such as node tasks, link tasks, full-graph tasks, etc.

Different subtasks usually require different task representations and different graph models.

Finally, the success of large language models is inseparable from context learning

(in-context learning)

achieved through prompt paradigms. In large language models, the prompt paradigm is usually a readable text description of the downstream task.

But for graph data that is unstructured and difficult to describe in words, how to design an effective graph prompt paradigm to achieve in-context learning is still an unsolved mystery.

Use the concept of "text diagram" to solve the problem

The following figure gives the overall framework of OFA:

Specifically, OFA's team solved the three main problems mentioned above through clever design.

To solve the problem of different graph data attributes and distributions, OFA unifies all graph data by proposing the concept of text graph

(Text-Attributed Graph, TAGs)

. Using text graphs, OFA describes the node information and edge information in all graph data using a unified natural language framework, as shown in the following figure:

Next, OFA uses a single LLM model to learn the representation of the text in all data to obtain its embedding vectors.

These embedding vectors will serve as input features for the graphical model. In this way, graph data from different domains will be mapped to the same feature space, making it feasible to train a unified GNN model.

OFA has collected 9 graph data sets of different sizes from different fields, including citation relationship graphs, Web link graphs, knowledge graphs, and molecular graphs, as shown in the following figure:

In addition, OFA proposes Nodes-of-Interest

（NOI）

Subgraph and NOI Prompt Node (NOI Prompt Node) to unify different subgraphs in the graph field Task type. Here NOI represents a set of target nodes participating in the corresponding task.

For example, in the node prediction task, the NOI refers to a single node that needs to be predicted; while in the link task, the NOI includes two nodes that need to predict the link. The NOI subgraph refers to a subgraph containing h-hop neighborhoods extended around these NOI nodes.

Then, the NOI prompt node is a newly introduced node type, directly connected to all NOIs.

Importantly, each NOI prompt node contains description information of the current task. This information exists in the form of natural language and is represented by the same LLM as the text graph.

Since the information contained in the nodes in the NOI will be collected by the NOI prompt node after passing the message of GNNs, the GNN model only needs to make predictions through the NOI prompt node.

In this way, all different task types will have a unified task representation. The specific example is shown in the figure below:

Finally, in order to realize in-context learning in the graph field, OFA introduces a unified prompt subgraph.

In a supervised k-way classification task scenario, this prompt subgraph contains two types of nodes: one is the NOI prompt node mentioned above, and the other represents k different categories. Class Node (Class Node).

The text of each category node will describe relevant information for this category.

NOI prompt nodes will be connected to all category nodes in one direction. The graph constructed in this way will be input into the graph neural network model for message passing and learning.

Finally, OFA will perform a two-classification task on each category node, and select the category node with the highest probability as the final prediction result.

Since category information exists in the cue subgraph, even if a completely new classification problem is encountered, OFA can directly predict without any fine-tuning by constructing the corresponding cue subgraph, thereby achieving zero-shot learning.

For a few-shot learning scenario, a classification task will include a query input graph and multiple support input graphs. OFA's prompt graph paradigm will associate the NOI prompt node of each support input graph with its corresponding category node. Connect, and at the same time connect the NOI prompt node of the query input graph to all category nodes.

The subsequent prediction steps are consistent with those described above. In this way, each category node will receive additional information from the support input graph, thereby achieving few-shot learning under a unified paradigm.

The main contributions of OFA are summarized as follows:

Unified graph data distribution: By proposing text graphs and using LLM to transform text information, OFA achieves distribution alignment and unification of graph data.

Uniform graph task form: Through NOI subgraphs and NOI prompt nodes, OFA achieves a unified representation of subtasks in various graph fields.

Unified graph prompting paradigm: By proposing a novel graph prompting paradigm, OFA realizes multi-scenario in-context learning in the graph field.

Super Generalization Ability

The article tested the OFA framework on 9 collected data sets. These tests covered ten different tasks in supervised learning scenarios. Including node prediction, link prediction and graph classification.

The purpose of the experiment is to verify the ability of a single OFA model to handle multiple tasks, in which the author compares the use of different LLM (OFA-{LLM}) and training a separate model for each task (OFA-ind-{LLM}) effect.

The comparison results are shown in the following table:

It can be seen that based on OFA’s powerful generalization ability, a separate graph model (OFA -st, OFA-e5, OFA-llama2-7b, OFA-llama2-13b) That is, it can have the same traditional separate training model on all tasks (GCN, GAT, OFA-ind-st ) Similar or better performance.

At the same time, using a more powerful LLM can bring certain performance improvements. The article further plots the representation of NOI prompt nodes for different tasks by the trained OFA model.

It can be seen that different tasks are embedded into different subspaces by the model, so that OFA can learn different tasks separately without affecting each other.

In the scenario of few samples and zero samples, OFA is used in ogbn-arxiv (reference graph) , FB15K237 (knowledge graph) and Chemble ( Use a single model to pre-train on molecular graphs), and test its performance on different downstream tasks and data sets. The results are as follows:

It can be seen that even in the zero-sample scenario, OFA can still achieve good results. Taken together, the experimental results well verify the powerful general performance of OFA and its potential as a basic model in the graph field.

For more research details, please refer to the original paper.

Address: https://www.php.cn/link/dd4729902a3476b2bc9675e3530a852chttps://github.com/ LechengKong/OneForAll

The above is the detailed content of The first universal framework in the graph field is here! Selected into ICLR\'24 Spotlight, any data set or classification problem can be solved. For more information, please follow other related articles on the PHP Chinese website!