Awesome Graph Self-Supervised Learning

A curated list for awesome self-supervised graph representation learning resources. Inspired by awesome-deep-vision, awesome-adversarial-machine-learning, awesome-deep-learning-papers, awesome-architecture-search, and awesome-self-supervised-learning.

Why Self-Supervised?

Self-Supervised Learning has become an exciting direction in AI community.

• Jitendra Malik: "Supervision is the opium of the AI researcher"
• Alyosha Efros: "The AI revolution will not be supervised"
• Yann LeCun: "self-supervised learning is the cake, supervised learning is the icing on the cake, reinforcement learning is the cherry on the cake"

Table of Contents

• Overview
• Training Strategy
• Contrastive Learning
  • Same-Scale Contrasting
    • Global-Global Contrasting
    • Context-Context Contrasting
    • Local-Local Contrasting
  • Corss-Scale Contrasting
    • Local-Global Contrasting
    • Local-Context Contrasting
    • Context-Global Contrasting
• Generative Learning
  • Graph Autoencoding
  • Graph Autoregression
• Predictive Learning
  • Node Property Prediction
  • Context-based Prediction
  • Self-Training
  • Domain Knowledge-based Prediction
• A Summary of Methodology Details
• A Summary of Implementation Details
• A Summary of Common Graph Datasets
• A Summary of Open-source Codes

Overview

We extend the concept of self-supervised learning, which first emerged in the fields of computer vision and natural language processing, to present a timely and comprehensive review of the existing SSL techniques for graph data. Specifically, we divide existing graph SSL methods into three categories: contrastive, generative, and predictive as shown below.

• Contrastive Learning: it contrasts the views generated by different data augmentation methods. The information about the differences and sameness between data-data pairs (inter-data) is used as self-supervision signals.
• Generative Learning: it focuses on the (intra-data) information embedded in the data, generally based on prtext tasks such as reconstruction, which exploit the attributes and structure of the data itself as self-supervision signals.
• Predictive Learning: it generally self-generates labels from graph data through some simple statistical analysis, or expert knowledge, and designs prediction-based pretext tasks based on the self-generated labels to handle the data-label relationship.

Training Strategy

Considering the relationship among bottleneck encoders, self-supervised pretext tasks, and downstream tasks, the training strategies can be divided into three categories: Pre-training and Fine-tuning (P&F), Joint Learning (JL), and Unsupervised Representation Learning (URL), with their detailed workflow shown below.

• Pre-train&Fine-tune (P&F): it first pre-trains the encoder with unlabeled nodes by the self-supervised pretext tasks. The pre-trained encoder’s parameters are then used as the initialization of the encoder used in supervised fine-tuning for downstream tasks.
• Joint Learning (JL): an auxiliary pretext task with self-supervision is included to help learn the supervised downstream task. The encoder is trained through both the pretext task and the downstream task simultaneously.
• Unsupervised Representation Learning (URL): it first pre-trains the encoder with unlabeled nodes by the self-supervised pretext tasks. The pre-trained encoder’s parameters are then frozen and used in the supervised downstream task with additional labels.

Contrastive Learning

A general framework for contrastive learning is shown below. The two contrasting components may be local, contextual, or global, corresponding to node-level (marked in red), subgraph-level (marked in green), or graph-level (marked in yellow) information in the graph. The contrastive learning can thus contrast two views (at the same or different scales), which leads to two categories of algorithm: (1) same-scale contrasting, including Local-Local (L-L) contrasting, Context-Context (C-C) contrasting, and Global-Global (G-G) contrasting; and (2) cross-scale contrasting, including Local-Context (L-C) contrasting, Local-Global (L-G) contrasting, and Context-Global (C-G) contrasting.

Global-Global Contrasting

• GraphCL: Graph Contrastive Learning with Augmentations.
  • Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen. NIPS 2020. [pdf] [code]
• IGSD: Iterative Graph Self-Distillation.
  • H. Zhang, S. Lin, W. Liu, P. Zhou, J. Tang, X. Arxiv 2020. [pdf]
• DACL: Towards Domain-Agnostic Contrastive Learning.
  • V. Verma, M.-T. Luong, K. Kawaguchi, H. Pham, andQ. V. Le. Arxiv 2020. [pdf]
• LCC: Label Contrastive Coding Based Graph Neural Network for Graph Classification.
  • Y. Ren, J. Bai, and J. Zhang. Arxiv 2021. [pdf] [code]
• CCGL: Contrastive Cascade Graph Learning.
  • X. Xu, F. Zhou, K. Zhang, and S. Liu. TKDE 2022. [pdf] [code]
• CSSL: Contrastive Self-Supervised Learning for Graph Classification.
  • J. Zeng and P. Xie. Arxiv 2020. [pdf]

Context-Context Contrasting

• GCC: Graph Contrastive Coding for Graph Neural Network Pre-training.
  • J. Qiu, Q. Chen, Y. Dong, J. Zhang, H. Yang, M. Ding, K. Wang, and J. Tang. KDD 2020. [pdf] [code]

Local-Local Contrasting

• GRACE: Deep Graph Contrastive Representation Learning.
  • Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang. Arxiv 2020. [pdf] [code]
• GCA: Graph Contrastive Learning with Adaptive Augmentation.
  • Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang. Arxiv 2020. [pdf] [code]
• GROC: Towards Robust Graph Contrastive Learning.
  • N. Jovanovi´c, Z. Meng, L. Faber, and R. Wattenhofer. Arxiv 2021. [pdf]
• SEPT: Socially-Aware Self-Supervised Tri-Training for Recommendation.
  • J. Yu, H. Yin, M. Gao, X. Xia, X. Zhang, and N. Q. V.Hung. Arxiv 2021. [pdf] [code]
• STDGI: Spatio-Temporal Deep Graph Infomax.
  • F. L. Opolka, A. Solomon, C. Cangea, P. Veliˇckovi´c, P. Li` o, and R. D. Hjelm. Arxiv 2019. [pdf]
• GMI: Graph Representation Learning via Graphical Mutual Information Maximization.
  • L. Yu, S. Pei, C. Zhang, L. Ding, J. Zhou, L. Li, and X. Zhang. WWW 2020. [pdf] [code]
• KS2L: Self-Supervised Smoothing Graph Neural Networks.
  • L. Yu, S. Pei, C. Zhang, L. Ding, J. Zhou, L. Li, and X. Zhang. Arxiv 2020. [pdf]
• CG3: Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning.
  • S. Wan, S. Pan, J. Yang, and C. Gong. Arxiv 2020. [pdf]
• BGRL: Bootstrapped Representation Learning on Graphs.
  • S. Thakoor, C. Tallec, M. G. Azar, R. Munos, P. Veliˇckovi´c, and M. Valko. Arxiv 2021. [pdf][code]
• SelfGNN: Self-supervised Graph Neural Networks without Explicit Negative Sampling.
  • Z. T. Kefato and S. Girdzijauskas. Arxiv 2021. [pdf] [code]
• HeCo: Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning.
  • X. Wang, N. Liu, H. Han, and C. Shi. Arxiv 2021. [pdf] [code]
• PT-DGNN: Pre-training on Dynamic Graph Neural Networks.
  • J. Zhang, K. Chen, and Y. Wang. Arxiv 2021. [pdf] [code]
• COAD: Coad: Contrastive Pretraining with Adversarial Fine-tuning for Zero-shot Expert Linking.
  • B. Chen, J. Zhang, X. Zhang, X. Tang, L. Cai, H. Chen, C. Li, P. Zhang, and J. Tang. Arxiv 2020. [pdf] [code]
• Contrast-Reg: Improving Graph Representation Learning by Contrastive Regularization.
  • K. Ma, H. Yang, H. Yang, T. Jin, P. Chen, Y. Chen, B. F. Kamhoua, and J. Cheng. Arxiv 2021. [pdf]
• C-SWM: Contrastive Learning of Structured World Models.
  • T. Kipf, E. van der Pol, and M. Welling. *Arxiv 2019. [pdf] [code]

Local-Global Contrasting

• DGI: Deep Graph Infomax.
  • P. Velickovic, W. Fedus, W. L. Hamilton, P. Li` o, Y. Bengio, and R. D. Hjelm. ICLR 2019. [pdf] [code]
• HDMI: Hdmi: High-order Deep Multiplex Infomax.
  • B. Jing, C. Park, and H. Tong. Arxiv 2021. [pdf]
• DMGI: Unsupervised Attributed Multiplex Network Embedding.
  • C. Park, D. Kim, J. Han, and H. Yu. AAAI 2020. [pdf] [code]
• MVGRL: Contrastive Multi-View Representation Learning on Graphs.
  • K. Hassani and A. H. K. Ahmadi. ICML 2020. [pdf] [code]
• HDGI: Heterogeneous Deep Graph Infomax.
  • Y. Ren, B. Liu, C. Huang, P. Dai, L. Bo, and J. Zhang. Arxiv 2019. [pdf] [code]

Local-Context Contrasting

• Subg-Con: Sub-graph Contrast for Scalable Self-Supervised Graph Representation Learning.
  • Y. Jiao, Y. Xiong, J. Zhang, Y. Zhang, T. Zhang, and Y. Zhu. Arxiv 2020. [pdf] [code]
• Cotext Prediction: Strategies for Pre-training Graph Neural Networks.
  • W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. S. Pande, and J. Leskovec. ICLR 2020. [pdf] [code]
• GIC: Leveraging Cluster-level Node Information for Unsupervised Graph Representation Learning.
  • C. Mavromatis and G. Karypis. Arxiv 2020. [pdf] [code]
• GraphLoG: Self-Supervised Graph-level Representation Learning with Local and Global Structure.
  • M. Xu, H. Wang, B. Ni, H. Guo, and J. Tang. OpenReview 2021. [pdf] [code]
• MHCN: Self-Supervised Multi-channel Hypergraph Convolutional Network for Social Recommendation.
  • J. Yu, H. Yin, J. Li, Q. Wang, N. Q. V. Hung, and X. Zhang. Arxiv 2021. [pdf] [code]
• EGI: Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization.
  • Q. Zhu, Y. Xu, H.Wang, C. Zhang, J. Han, and C. Yang. Arxiv 2020. [pdf] [code]

Context-Global Contrasting

• MICRO-Graph: Motif-Driven Contrastive Learning of Graph Representations.
  • S. Zhang, Z. Hu, A. Subramonian, and Y. Sun. Arxiv 2020. [pdf] [code]
• InfoGraph: Unsupervised and Semi-Supervised Graph-level Representation Learning via Mutual Information Maximization.
  • F. Sun, J. Hoffmann, V. Verma, and J. Tang. ICLR 2020. [pdf] [code]
• SUGAR: Subgraph Neural Network with Reinforcement Pooling and Self-Supervised Mutual Information Mechanism.
  • Q. Sun, H. Peng, J. Li, J. Wu, Y. Ning, P. S. Yu, and L. He. Arxiv 2021. [pdf] [code]
• BiGI: Bipartite Graph Embedding via Mutual Information Maximization.
  • J. Cao, X. Lin, S. Guo, L. Liu, T. Liu, and B. Wang. WSDM 2021. [pdf] [code]
• HTC: Graph Representation Learning by Ensemble Aggregating Subgraphs via Mutual Information Maximization.
  • C. Wang and Z. Liu. Arxiv 2021. [pdf]
• DITNet: Drug Target Prediction using Graph Representation Learning via Substructures Contrast.
  • S. Cheng, L. Zhang, B. Jin, Q. Zhang, and X. Lu. Preprints 2021. [pdf] [code]

Generative Learning

Graph Autoencoding

• GraphMAE: Self-supervised Masked Graph Autoencoders
  • Z. Hou, X. Liu, Y. Cen, Y. Dong, H. Yang, C. Wang, and J. Tang. KDD 2022 [pdf] [code]
• Graph Completion: When Does Self-Supervision Help Graph Convolutional Networks?
  • Y. You, T. Chen, Z. Wang, and Y. Shen. PMLR 2020. [pdf] [code]
• Node Attribute Masking: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
  • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]
• Edge Attribute Masking: Strategies for Pre-training Graph Neural Networks.
  • W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. S. Pande, and J. Leskovec. ICLR 2020. [pdf] [code]
• Node Attribute and Embedding Denoising: Graph-based Neural Network Models with Multiple Self-Supervised Auxiliary Tasks.
  • F. Manessi and A. Rozza. Arxiv 2020. [pdf]
• Adjacency Matrix Reconstruction: Self-Supervised Training of Graph Convolutional Networks.
  • Q. Zhu, B. Du, and P. Yan. Arxiv 2020. [pdf]
• Graph Bert: Only Attention is Needed for Learning Graph Representations.
  • J. Zhang, H. Zhang, C. Xia, and L. Sun. Arxiv 2020. [pdf] [code]
• Pretrain-Recsys: Pretraining Graph Neural Networks for Cold-start Users and Items Representation.
  • B. Hao, J. Zhang, H. Yin, C. Li, and H. Chen. WSDM 2021. [pdf] [code]
• SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks.
  • B. Fatemi, L. E. Asri, and S. M. Kazemi. Arxiv 2021. [pdf] [code]
• G-BERT: Pre-Training of Graph Augmented Transformers for Medication Recommendation.
  • J. Shang, T. Ma, C. Xiao, and J. Sun. Arxiv 2019. [pdf] [code]

Graph Autoregression

• GPT-GNN: Generative Pre-training of Graph Neural Networks.
  • Z. Hu, Y. Dong, K. Wang, K. Chang, and Y. Sun. KDD 2020. [pdf] [code]

Predictive Learning

A comparison of the predictive learning is shown below. The predictive method generally self-generates labels from graph data and then designs prediction-based pretext tasks based on the self-generated labels. Categorized by how the labels areobtained, we summarize predictive learning methods forgraph data into four categories:

• Node Property Prediction: it pre-calculates the node properties, such as node degree and used them as self-supervised labels.
• Context-based Prediction: the local or global contextual information in the graph, such as the shortest path length between nodes can be extracted as labels to help with self-supervised learning.
• Self-Training: it applies algorithms such as unsupervised clustering to obtain pseudo-labels and then updates the pseudo-label set of the previous stage based on the prediction results or losses.
• Domain Knowledge-based Prediction: the domain knowledge, such as expert knowledge or specialized tools, can be used in advance to obtain informative labels.

Node Property Prediction

• Node Property Prediction: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
  • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]

Context-based Prediction

• S2GRL: Self-Supervised Graph Representation Learning via Global Context Prediction.
  • Z. Peng, Y. Dong, M. Luo, X.-M. Wu, and Q. Zheng. Arxiv 2020. [pdf]
• PairwiseDistance: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
  • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]
• PairwiseAttsim: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
  • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]
• Distance2Cluster: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
  • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]
• EdgeMask: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
  • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]
• TopoTER: Unsupervised Learning of Topology Transformation Equivariant Representations.
  • X. Gao, W. Hu, and G.-J. Qi. OpenReview 2021. [pdf]
• Centrality Score Ranking: Pretraining Graph Neural Networks for Generic Structural Feature Extraction.
  • Z. Hu, C. Fan, T. Chen, K.-W. Chang, and Y. Sun. Arxiv 2019. [pdf]
• Meta-path prediction: Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs.
  • D. Hwang, J. Park, S. Kwon, K. Kim, J. Ha, and H. J. Kim. NIPS 2020. [pdf] [code]
• SLiCE: Self-Supervised Learning of Contextual Embeddings for Link Prediction in Heterogeneous Networks.
  • P. Wang, K. Agarwal, C. Ham, S. Choudhury, and C. K. Reddy. Arxiv 2020. [pdf] [code]
• Distance2Labeled: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
  • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]
• Distance2Labeled: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
  • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]
• HTM: Hop-count based Self-Supervised Anomaly Detection on Attributed Networks.
  • T. Huang, Y. Pei, V. Menkovski, and M. Pechenizkiy. Arxiv 2021. [pdf]

Self-Training

• Multi-stage Self-training: Deeper insights into Graph Convolutional Networks for Semi-Supervised Learning.
  • Q. Li, Z. Han, and X. Wu. AAAI 2018. [pdf] [code]
• Node Clustering and Partitioning: When Does Self-Supervision Help Graph Convolutional Networks.
  • Y. You, T. Chen, Z. Wang, and Y. Shen. PMLR 2020. [pdf] [code]
• CAGAN: Cluster-Aware Graph Neural Networks for Unsupervised Graph Representation Learning.
  • Y. Zhu, Y. Xu, F. Yu, S. Wu, and L. Wang. Arxiv 2020. [pdf]
• M3S: Multi-stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labeled Nodes.
  • K. Sun, Z. Lin, and Z. Zhu. AAAI 2020. [pdf] [code]
• Cluster Preserving: Pretraining Graph Neural Networks for Generic Structural Feature Extraction.
  • Z. Hu, C. Fan, T. Chen, K.-W. Chang, and Y. Sun. Arxiv 2019. [pdf]
• SEF: Self-Supervised Edge Features for Improved Graph Neural Network Training.
  • A. Sehanobish, N. G. Ravindra, and D. van Dijk. Arxiv 2020. [pdf][code]

Domain Knowledge-based Prediction

• Contextual Molecular Property Prediction: Self-Supervised Graph Transformer on Large-Scale Molecular Data.
  • Y. Rong, Y. Bian, T. Xu, W. Xie, Y. Wei, W. Huang, and J. Huang. NIPS 2020. [pdf] [code]
• Graph-level Motif Prediction: Self-Supervised Graph Transformer on Large-scale Molecular Data.
  • Y. Rong, Y. Bian, T. Xu, W. Xie, Y. Wei, W. Huang, and J. Huang. NIPS 2020. [pdf] [code]
• DrRepair: Graph-based, Self-Supervised Program Repair from Diagnostic Feedback.
  • M. Yasunaga and P. Liang. PMLR 2020. [pdf] [code]

A summary of all the surveyed works is presented below.

A Summary of Methodology Details

About Graph Property, Pretext Task, Data Augmentation, Objective Function, Training Strategy, and Year of publication.

A Summary of Implementation Details

About Task Level, Evaluation Metric, and Evaluation Datasets.

A Summary of Common Graph Datasets

About category, graph number, node number per graph, edge number per graph, dimensionality of node attributes, class number, and citation papers.

A Summary of Open-source Codes

Contribute

If you would like to help contribute this list, please feel free to contact me or add pull request with the following Markdown format:

- Paper Name. 
  - Author List. *Conference Year*. [[pdf]](link) [[code]](link)

This is a Github Summary of our Survey. If you find this file useful in your research, please consider citing:

@article{wu2021self,
  title={Self-supervised Learning on Graphs: Contrastive, Generative, or Predictive},
  author={Wu, Lirong and Lin, Haitao and Tan, Cheng and Gao, Zhangyang and Li, Stan Z},
  journal={IEEE Transactions on Knowledge and Data Engineering},
  year={2021},
  publisher={IEEE}
}

Feedback

If you have any issue about this work, please feel free to contact me by email:

• Lirong Wu: [email protected]