Graph Foundational Model

Pre-training for Graph Neural Networks

Part – 3

• Intro to Graph Pre-Training
• Contrastive Pre-Training
• Generative Pre-Training

Issues in Supervised Deep GNN

• Requires Large Amount of Labelled Data
• Task-specific labeled data can be extremely scarce.
• Data labeling 🡪 resource and time intensive
• GNNs overfit to small training data
• Low out-of-distribution generalization ability of the trained models.
• Graph data from real-world applications often contain out-of-distribution samples
• GNNs extrapolate poorly

GNN Pre-training

• Effective Solution: Pre-train and Transfer Learning/Fine-tuning
• Train a GNN model with a massive corpus of pre-training graphs.
• Utilize the pre-trained GNN model as initialization
• Fine-tune the model parameters based on the specific downstream task.

Challenges in GNN Pre-training

• Requires large number of labeled pre-training datasets that are from the same domain as the downstream task.
• Carefully select pretraining task that are correlated with the downstream task of interest
• Naive Pre-training leads to negative transfer

Graph Foundation Model

Source: Liu et al.

Self-Supervised GNN Pre-training

• Contrastive SSL Pre-training
• Constructs the supervised signals at the inter-graph level and learns the representation by contrasting with other graphs
• Generative SSL Pre-training
• Focuses on reconstructing the original graph at the intra-graph level.

Pre-Training: Generative Strategy

Graph Reconstruction

Source: Liu et al.

Property Prediction

Pre-Training: Contrastive Strategy

• maximize the MI between positive pairs

(e.g., augmented instances of the same object)

and minimize the MI between negative samples

(e.g., instances derived from different objects

Source: Liu et al.

Strategies for Pre-training Graph Neural Networks, Weihua Hu et al. ICLR 2020

Strategies for Pre-training Graph Neural Networks

Generative SSL Pre-training

Key Insight

• Develop an effective (Self-supervised) strategy for pre-training GNNs.
• Pre-train an expressive GNN at the level of individual nodes as well as entire graphs.
• Learn useful local and global representations simultaneously

Pretraining GNN

• Pre-training has been hugely successful in computer vision and natural language processing.
• Let’s consider pre-training GNNs!
• Q1. How effective is pre-training GNNs?
• Q2. What is the effective pre-training strategy?

How Effective is Pre-training GNNs

• Naive strategy - Multi-task supervised pre-training

How Effective is Pre-training GNNs

• Naive strategy - Limited performance improvement on downstream tasks.(Negative Transfer)

What is Effective Pretraining Strategy

• Key idea: Pre-train both node and graph embeddings.

What is Effective Pre-training Strategy

• Key idea: Pre-train both node and graph embeddings.
• GNN can capture domain-specific knowledge of both local and global structure.

Proposed Pre-training Method

Node-level : Attribute Masking

• Mask node/edge attributes
• Use GNN to generate node embeddings
• Use the embeddings to predict masked attributes.

Node-level : Attribute Masking

• Through solving the masked attribute prediction task, a GNN is forced to learn domain knowledge, .e.g., chemical rules.
• Learning the regularities of the node/edge attributes distributed over graph structure.

Node-level : Context Prediction

• For each graph, sample one center node v.
• Extract neighborhood and context graphs for v.
• (K-Hop) Neighborhood Graphs: all nodes and edges that are at most K-hops away from v.
• Context Graph: subgraph that is between r₁-hops and r₂-hops away from v (i.e., it is a ring of width r₁ - r₂)

Node-level : Context Prediction

• Goal - Predict context graph of center node v using node embedding h_v^(K), generated by K-layer GNN.
• Use GNNs to encode neighborhood and context graphs into vectors/embedding
• Intuition: Subgraphs that are surrounded by similar contexts are semantically similar. [word2vec model, Mikolov et al. NIPS 2013].

Node-level : Context Prediction

• Use negative sampling to jointly learn the main GNN and the context GNN.
• Maximize/minimize the inner product between true/false (neighborhood, context) pairs.

​

• Positive neighborhood-context pair, let v = v′ and G = G′.
• Negative neighborhood-context pair, randomly sample v′ from a randomly chosen graph G′.

Graph-level : Supervised Attribute Prediction

• Multi-task supervised training on many relevant labels.

Overall Strategy

Experimental Results

• Avoids negative transfer.
• Significantly improve the performance.