Graph Machine Learning under Distribution Shifts:

Adaptation, Generalization and Extension to LLM

IJCAI 2024, Jeju

Survey, Paper Collections

Speakers

Xin Wang Tsinghua University, China

Xin Wang is currently an Associate Professor at the Department of Computer Science and Technology, Tsinghua University. He got both his Ph.D. and B.E degrees in Computer Science and Technology from Zhejiang University, China. He also holds a Ph.D. degree in Computing Science from Simon Fraser University, Canada. His research interests include multimedia intelligence, machine learning and its applications in multimedia big data analysis. He has published over 150 high-quality research papers in top journals and conferences including IEEE TPAMI, IEEE TKDE, ACM TOIS, ICML, NeurIPS, ACM KDD, ACM Web Conference, ACM SIGIR and ACM Multimedia etc., winning three best paper awards. He is the recipient of 2020 ACM China Rising Star Award, 2022 IEEE TCMC Rising Star Award and 2023 DAMO Academy Young Fellow.

Haoyang Li Cornell University, USA

Haoyang Li is currently a postdoc researcher at Weill Cornell Medicine of Cornell University. He received his Ph.D. from the Department of Computer Science and Technology of Tsinghua University in 2023. He received his B.E. from the Department of Computer Science and Technology of Tsinghua University in 2018. His research interests are mainly in machine learning on graphs and out-of-distribution generalization. He has published high-quality papers in prestigious journals and conferences, e.g., IEEE TKDE, ACM TOIS, NeurIPS, ICLR, ACM KDD, ACM Web Conference, AAAI, IJCAI, ACM Multimedia, IEEE ICDE, IEEE ICDM, etc., winning one best paper award.

Zeyang Zhang Tsinghua University, China

Zeyang Zhang received his B.E. from the Department of Computer Science and Technology, Tsinghua University in 2020. He is a Ph.D. candidate in the Department of Computer Science and Technology of Tsinghua University. His main research interests focus on automated machine learning, out-of-distribution generalization and large language models. He has published several papers in prestigious conferences, e.g., NeurIPS, AAAI, etc.

Wenwu Zhu Tsinghua University, China

Wenwu Zhu is currently a Professor in the Department of Computer Science and Technology at Tsinghua University, the Vice Dean of National Research Center for Information Science and Technology, and the Vice Director of Tsinghua Center for Big Data. Prior to his current post, he was a Senior Researcher and Research Manager at Microsoft Research Asia. He was the Chief Scientist and Director at Intel Research China from 2004 to 2008. He worked at Bell Labs New Jersey as Member of Technical Staff during 1996-1999. He received his Ph.D. degree from New York University in 1996.

His research interests include graph machine learning, curriculum learning, data-driven multimedia, big data. He has published over 400 referred papers, and is inventor of over 80 patents. He received ten Best Paper Awards, including ACM Multimedia 2012 and IEEE Transactions on Circuits and Systems for Video Technology in 2001 and 2019.

He serves as the EiC for IEEE Transactions on Circuits and Systems for Video Technology, the EiC for IEEE Transactions on Multimedia (2017-2019) and the Chair of the steering committee for IEEE Transactions on Multimedia (2020-2022). He serves as General Co-Chair for ACM Multimedia 2018 and ACM CIKM 2019. He is an AAAS Fellow, IEEE Fellow, ACM Fellow, SPIE Fellow, and a member of Academia Europaea.

Tutorial Description

Graph machine learning has been extensively studied in both academia and industry. Although booming with a vast number of emerging methods and techniques, most of the literature is built on the in-distribution (I.D.) hypothesis, i.e., testing and training graph data are sampled from the identical distribution. However, this I.D. hypothesis can hardly be satisfied in many real-world graph scenarios where the model performance substantially degrades when there exist distribution shifts between testing and training graph data. To solve this critical problem, several advanced graph machine learning techniques which go beyond the I.D. hypothesis, have made great progress and attracted ever-increasing attention from the research community. This tutorial is to disseminate and promote the recent research achievement on graph out-of-distribution adaptation, graph out-of-distribution generalization, and large language models for tackling distribution shifts, which are exciting and fast-growing research directions in the general field of machine learning and data mining. We will advocate novel, high-quality research findings, as well as innovative solutions to the challenging problems in graph machine learning under distribution shifts and the applications on graphs. This topic is at the core of the scope of IJCAI, and is attractive to machine learning as well as data mining audience from both academia and industry.


Tutorial Outline

To the best of our knowledge, this tutorial is the first to systematically and comprehensively discuss graph machine learning under distribution shifts, with a great potential to draw a large amount of interests in the community. The tutorial is planned for 1/4 day and organized into 4 sections.

  • The research and industrial motivation for graph machine learning under distribution shifts
  • Graph Adaptation under distribution shifts
  • Graph Generalization under distribution shifts
  • Graph LLM
  • Discussions and future directions


    • Target Audience and Prerequisites

    This tutorial will be highly accessible to the whole machine learning and data mining community, including researchers, students and practitioners who are interested in graph adaptation, graph generalization, graph LLM and their applications in graph-related tasks. The tutorial will be self-contained and designed for introductory and intermediate audiences. No special prerequisite knowledge is required to attend this tutorial.


    Motivation, Relevance and Rationale

    Graph machine learning under distribution shifts is becoming a hot research topic in both academia and industry. This tutorial is to disseminate and promote the recent research achievements on graph machine learning under distribution shifts as well as their applications on graphs, which is an exciting and fast-growing research direction in the general field of machine learning and graph neural networks. We will advocate novel, high-quality research findings, as well as innovative solutions to the challenging problems in graph machine learning under distribution shifts. This topic is at the core of the scope of IJCAI and is attractive to IJCAI audience from both academia and industry.


    Tutorial Overview

    We characterize graph machine learning under distribution shifts as those that not only demonstrate competence but also integrate fundamental elements. These elements encompass graph adaptation, graph generalization, and graph LLM, specifically in relation to GNNs. We posit that embedding these characteristics in the design phase of graph ML models will greatly enhance their trustworthiness, thereby fostering their industrial application and widespread use.

    In this tutorial, we discuss promising solutions to graph machine learning under distribution shifts from three aspects:

    Graph Adaptation under distribution shifts

    As graphs evolve over time, encountering shifts in their underlying distributions, traditional graph ML models face challenges in maintaining performance and adaptability. In response, the recent research focus has shifted towards developing adaptive techniques that can seamlessly adjust to these distribution shifts while preserving model effectiveness. One classic of methods are graph neural architecture search methods, which empower algorithms to autonomously explore and discover effective graph architectures tailored to specific tasks. By automating the design process, graph neural architecture search methods enable the creation of models optimized for adaptability under distribution shifts, fostering more resilient systems. Another classic of methods are graph continual learning methods, which complement these efforts by focusing on models' ability to learn and adapt over time. Rather than treating data as static entities, continual learning frameworks facilitate ongoing model updates and refinement, enabling adaptation to evolving distributions without catastrophic forgetting and advancing the performance of graph models in dynamic real-world scenarios. We will discuss works which aim to propose new approaches to adapt graph ML models under distribution shifts, including two types of representative methods: graph neural architecture search and graph continual learning.

    Graph Generalization under distribution shifts

    Besides Out-of-distribution generalized graph model, some works focus on exploiting training schemes with tailored optimization objectives and constraints to promote OOD generalization on graphs, including graph invariant learning, graph adversarial training, and graph self-supervised learning. Some graph invariant learning methods are built upon the invariance principle to address the OOD generalization problem from a principle way, which aim to exploit the invariant relationships between features and labels across different distributions while disregarding the variant spurious correlations. Some graph adversarial training and graph self-supervised learning methods have also been demonstrated to improve graph model robustness against distribution shifts and OOD generalization ability. We will give an introduction to these graph training strategies and introduce some recent approaches for handling OOD generalization.

    Graph LLM

    Large language models (LLMs) have demonstrated remarkable capabilities in in-context learning and few-shot learning, showcasing their ability to generalize across diverse tasks without extensive training or fine-tuning. This inherent adaptability has drawed the interest of researchers in graph-related fields. Numerous studies have sought to integrate LLMs and Graph Neural Networks (GNNs) to develop versatile graph models capable of handling a wide array of data distributions. Some approaches leverage LLMs to enrich the feature representations used by GNNs, enhancing their ability to capture nuanced information from graph data. Conversely, other methodologies utilize GNNs to empower LLMs with structural knowledge, enabling them to effectively tackle graph-related tasks by encoding graph structures into their learning process. This relationship between LLMs and GNNs holds promise for advancing the state-of-the-art in graph representation learning and opens avenues for more robust and adaptable graph-based applications under distribution shifts. We will introduce methods which focus on combining the abilities of large language models to enhance the graph out-of-distribution generalization capability under distribution shifts, including LLMs-enhanced GNNs and GNNs-enhanced LLMs, enabling better in-context learning and few-shot learning abilities across graph distributions in real-world scenarios.

    The tutorial will be presented lively. However, in case of any technical problems, we may also provide a pre-recorded video for the tutorial.

    Updated on Apr, 2024