Causal Representation Learning in Visual Understanding

Carnegie Mellon University

Mohamed bin Zayed University of Artificial Intelligence

Guangyi Chen

April 22^nd, 2025

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

2

How Do We Solve Visual Tasks?

Segment each part of the cat.

Compare it with other images.

A cute cat on the blanket

Caption it to describe this image.

Identify what it is and avoid collisions.

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

3

Representation Matters

A cute cat on the blanket

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

4

What Is Good Representation

• Compact
• Interpretable
• Meaningful
• Transferable
• Discriminative
• Controllable

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

5

What Is Good Representation

Deep Learning

• Compact
• Interpretable
• Meaningful
• Transferable
• Discriminative
• Controllable

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

6

What Is Good Representation

• Compact
• Interpretable
• Meaningful
• Transferable
• Discriminative
• Controllable

How to learn the representation that is

interpretable, transferable, and controllable?

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

7

Causal Representation Learning

Color

Pose

View

Generation

Process

• Causal representation learning aims to invert the data generation process to identify the ground-truth causal factor. 

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

8

Identifiability

​

Observed data distribution

Parameter (representation) space

Color

Pose

View

Component-wise Identifiability

[ ]

[ ]

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

9

Representation Disentanglement

​

Color

Pose

View

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

10

A Simple Linear Case: X=AZ

​

• The non-Gaussian provides “auxiliary information” to identify the true latent variables

Latent variable (z)

Observation (x)

Latent variable (z)

Observation (x)

Linear Gaussian

Linear Non-Gaussian

Learning causal representation is non-trivial

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

11

Investigate and demonstrate the practical contributions of causal representation to visual understanding

Real-world Application

Explore methods for learning causal representations and establish their identification conditions

Theoretical Foundation

Causal Representation Learning

My Research

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

12

Investigate and demonstrate the practical contributions of causal representation to visual understanding

Real-world Application

Explore methods for learning causal representations and establish their identification conditions

Theoretical Foundation

Theoretical Foundation

​

Causal Representation Learning

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

13

Review of Causal Representation Learning

Data Generation Process

Color

Pose

View

Cartoon

Causal Representation Learning

Equivalent

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

14

Sufficient Change Principle

​

Data Generation Process

Identification Condition

Kong et al. Partial disentanglement for domain adaptation. ICML, 2022

Color

Pose

View

Cartoon

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

15

Sufficient Change from Temporal Information

​

Yao., Chen., and Zhang. Temporally disentangled representation learning. NeurIPS, 2022

Historical State

Current State

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

16

An Example of the Temporal Case

​

• For the following mass-spring dynamic system, five objects can be disentangled in the scene, and thus the relations among these objects can be identified.

Yao., Chen., and Zhang. Temporally disentangled representation learning. NeurIPS, 2022

Observed video of the mass-spring system

Recovered variables and the corresponding relations

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

17

Extension under Known Non-stationary

​

• If observed, the non-stationarity can serve as an auxiliary variable, providing change information, whether it occurs in the transition function or the observation.

Yao., Chen., and Zhang. Temporally disentangled representation learning. NeurIPS, 2022

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

18

Extension under Unknown Non-Stationary

​

• When the nonstationary information cannot be observed, we can leverage extra assumptions/priors to help estimate the non-stationarity, e.g., the Markov assumption.

Song et al. Temporally Disentangled Representation Learning under Unknown Nonstationarity. NeurIPS, 2023.

A video clip of a mouse motion

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

19

Results under Unknown Non-Stationary

​

• We demonstrate that, under the Markov assumption, it is possible to effectively estimate unseen non-stationarity (e.g., motion phases) and distinguish active and inactive phases.

Song et al. Temporally Disentangled Representation Learning under Unknown Nonstationarity. NeurIPS, 2023.

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

20

Extension under Non-Invertibility

​

• The generation process may be non-invertible, caused by occlusion. To solve this problem, we apply the sequential encoder to leverage the context to recover the lost information.

Chen et al. CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process. ICML, 2024.

A traffic video example illustrates non-invertibility caused by occlusion.

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

21

Results under Non-Invertibility

​

• Compared to the original step encoder, the sequential variant leverages contextual information to effectively recover details lost due to occlusion.

Chen et al. CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process. ICML, 2024.

Sequential Encoder

MSE: 0.045

Step Encoder

MSE: 0.010

Ground Truth

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

22

Extension under Instantaneous Dependency

​

​

• All aforementioned methods assume the absence of instantaneous dependencies, which may not hold true when there are solid relations.

The relations among human joints are instantaneously dependent.

Li et al. On the Identification of Temporally Causal Representation with Instantaneous Dependence. ICLR, 2025.

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

23

Sparsity Principle

​

Li et al. On the Identification of Temporally Causal Representation with Instantaneous Dependence. ICLR, 2025.

Identification Condition

Sparse

Constraint

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

24

Results under Instantaneous Dependency

​

​

• In human motion scenarios where instantaneous dependencies exist among latent variables, methods with sparsity constraints can achieve better prediction accuracy.

Li et al. On the Identification of Temporally Causal Representation with Instantaneous Dependence. ICLR, 2025.

w/o Sparsity

MSE: 0.0102

with Sparsity

MSE: 0.0093

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

25

Generative Learning Framework

Encoder

Encoder

Encoder

…

…

…

Yao., Chen., and Zhang. Temporally disentangled representation learning. NeurIPS, 2022

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

26

Generative Learning Framework

Encoder

Encoder

Encoder

…

…

…

• Conditional Independence — Prior Network

We add a KL divergence loss between the posterior and a conditionally independent prior. This prior is estimated by an invertible flow model, which decomposes the prior into a Gaussian noise and a Jacobian of the flow.

flow

Gaussian noise

Jacobin of flow

Yao., Chen., and Zhang. Temporally disentangled representation learning. NeurIPS, 2022

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

27

Generative Learning Framework

…

…

…

• Extension for Unknown Non-stationary — Autoregressive Hidden Markov Module

We apply a sequential model, like LSTM, to estimate the unknown non-stationary domain index, and use it as a condition of the flow network.

Song et al. Temporally Disentangled Representation Learning under Unknown Nonstationarity. NeurIPS, 2023.

Encoder

Encoder

Encoder

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

28

Generative Learning Framework

…

…

• Extension for Non-invertibility — Sequential Encoder:

To recover information lost at the current step, we replace the original step encoder with a sequential encoder that incorporates temporal context.

…

Chen et al. CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process. ICML, 2024.

Encoder

Encoder

Encoder

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

29

Generative Learning Framework

…

…

…

Li et al. On the Identification of Temporally Causal Representation with Instantaneous Dependence. ICLR, 2025.

Encoder

Encoder

Encoder

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

30

Investigate and demonstrate the practical contributions of causal representation to visual understanding

Real-world Application

Explore methods for learning causal representations and establish their identification conditions

Theoretical Foundation

Real-World Application

​

Causal Representation Learning

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

31

Potential Contributions of CRL

Transferable

Interpretable

Reasoning-capable

Controllable

Q: Would the accident still happen if fewer vehicles were on the road?

A: Yes, the road is not congested in the first place.

A dog with a Golden/Diamond/Flower crown holding a sign with 2025 printed on it

adapt

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

32

Causal Representation for Transfer Learning

Q: Would the accident still happen if fewer vehicles were on the road?

A: Yes, the road is not congested in the first place.

A dog with a Golden/Diamond/Flower crown holding a sign with 2025 printed on it

adapt

Transferable

Interpretable

Reasoning-capable

Controllable

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

33

Causal Representation for Transfer Learning

adapt

• In real-world applications, the training and testing data may follow different distributions, requiring the adaptation of the learned model from the training domain to the testing one.

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

34

Multi-Source Domain Adaptation

Source Domains

Target Domain

Training

Stage

Testing

Stage

Labeled Data

Unlabeled Data

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

35

Causal Representation vs. Invariant Representation

Kong et al. Partial disentanglement for domain adaptation. ICML, 2022

• Most domain adaptation methods learn invariant representations, while our CRL framework models the joint distribution to also leverage the bias information.

Invariant Representation

Causal Representation

Content

Style

D1

D2

Contrastive

Loss

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

36

Kong et al. Partial disentanglement for domain adaptation. ICML, 2022

Identify invariance by comparison

Identify the style part by the sufficient change

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

37

Experimental Results

Kong et al. Partial disentanglement for domain adaptation. ICML, 2022

• Compared to methods that learn invariants, our causal representation learning approach achieves superior adaptation performance.

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

38

Extensions under Label-Shift

Li et al. Subspace identification for multi-source domain adaptation. NeurIPS, 2023.

• Further consider the relations among latent variables.
• Handle all types of distribution shifts

Ng et al. A General Representation-Based Approach to Multi-Source Domain Adaptation. In submission.

Extensions for a General Case

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

39

Experimental Results

• Considering more general and complex cases can further enhance performance.

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

40

Extension for One-Sample Extrapolation

Kong.*, Chen.* et al. Towards Understanding Extrapolation: a Causal Lens. NeurIPS, 2024

Domain Adaptation

Extrapolation

• Require the entire target distribution.
• The target domain overlaps with the source domain's support.

• Access even just one sample
• Target data may be outside the training domain support

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

41

Extension for One-Sample Extrapolation

Kong*, Chen*, et al. Towards Understanding Extrapolation: a Causal Lens. NeurIPS, 2024

• Dense shifts require mild off-support deviations; sparse shifts demand the preservation of invariants.  It motivates aligning invariant variables and applying the sparsity constraint for minimal change.

Dense Shifts (camera view)

Sparse Shifts (background)

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

42

Experimental Results

Evaluation of the invariant variables' alignment

Evaluation of the sparsity constraint

Kong*, Chen*, et al. Towards Understanding Extrapolation: a Causal Lens. NeurIPS, 2024

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

43

Causal Representation for Visual Analysis

Q: Would the accident still happen if fewer vehicles were on the road?

A: Yes, the road is not congested in the first place.

A dog with a Golden/Diamond/Flower crown holding a sign with 2025 printed on it

adapt

Controllable

Transferable

Interpretable

Reasoning-capable

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

44

Natural Hierarchies of Visual Concepts

Kong, Chen, et al. Learning Discrete Concepts in Latent Hierarchical Models. NeurIPS, 2024

• Concept Learning: Identifying the discrete variables in the hierarchical model.

High-level concepts:

cat, cute, …

Low-level concepts:

eyes, fur, blanket, …

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

45

Understand Diffusion Model with Hierarchical Model

Kong, Chen, et al. Learning Discrete Concepts in Latent Hierarchical Models. NeurIPS, 2024

• We illustrate that diffusion models' multi-level noise addition enables the model to estimate the concept hierarchy.
• Low-level concepts are learned at low noise levels, whereas high-level concepts are learned at high noise levels.

Noise drowns low-level information.

Text supplies missing concepts.

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

46

Experimental Justification

Kong, Chen, et al. Learning Discrete Concepts in Latent Hierarchical Models. NeurIPS, 2024

• Inject high-level and low-level concepts in hierarchical order to ensure faithful representation—violating this order leads to the issues.

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

47

Identify Hierarchies with ``Intervening’’

Kong, Chen, et al. Learning Discrete Concepts in Latent Hierarchical Models. NeurIPS, 2024

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

48

Causal Representation for Visual Reasoning

A dog with a Golden/Diamond/Flower crown holding a sign with 2025 printed on it

Q: Would the accident still happen if fewer vehicles were on the road?

A: Yes, the road is not congested in the first place.

adapt

Interpretable

Reasoning-capable

Controllable

Transferable

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

49

Video Reasoning

Chen et al. LLCP: Learning Latent Causal Processes for Reasoning-based Video Question Answer. ICLR 2024.

• Video reasoning aims to answer the neural language reasoning questions based on the video content, whose challenge lies in understanding the latent causal process.

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

50

Comparison with Existing Methods

Chen et al. LLCP: Learning Latent Causal Processes for Reasoning-based Video Question Answer. ICLR 2024.

• Once we identify the causal dynamics, we can efficiently conduct video reasoning (such as attribution and counterfactual questions) as a causal inference process.

• Formulate VideoQA as cross-modality matching.
• Rely on question-answer (QA) pairs, Learn relations

• Don’t require the QA pairs.
• Can efficiently answer the attribution and counterfactual questions

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

51

Overall Framework

Chen et al. LLCP: Learning Latent Causal Processes for Reasoning-based Video Question Answer. ICLR 2024.

• In the training, we learn the latent causal process from normal videos by self-supervision.
• During inference, we first identify the root cause and use it to select the answer.

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

52

Experimental Showcase

Chen et al. LLCP: Learning Latent Causal Processes for Reasoning-based Video Question Answer. ICLR 2024.

• By identifying the root cause with the learned temporal dynamic, we can easily answer the attribution questions.

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

53

Causal Representation for Controllable Visual Generation

Q: Would the accident still happen if fewer vehicles were on the road?

A: Yes, the road is not congested in the first place.

A dog with a Golden/Diamond/Flower crown holding a sign with 2025 printed on it

adapt

Reasoning-capable

Interpretable

Transferable

Controllable

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

54

Controllable Text-to-Image Generation

Xie et al. Aligning Atomic Vision Language Concepts for Controllable Image Generation. In submission.

A farmer with red clothes walks through his garden at sunrise with a cat.

A farmer with blue clothes walks through his garden at sunrise with a cat.

The existing generative model cannot produce precise control.

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

55

Controllable Text-to-Image Generation

Xie et al. Aligning Atomic Vision Language Concepts for Controllable Image Generation. In submission.

A farmer with red clothes walks through his garden at sunrise with a cat.

A farmer with blue clothes walks through his garden at sunrise with a cat.

Adapt generation with minimal change

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

56

Intuition and Framework

Xie et al. Aligning Atomic Vision Language Concepts for Controllable Image Generation. In submission.

Learnable

Query

Conditional generation

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

57

Generated Examples

Xie et al. Aligning Atomic Vision Language Concepts for Controllable Image Generation. In submission.

Golden

Diamond

Flower

DALLE3

Ours

• The comparison between DALLE3 and our model with prompt “A dog with a Golden/Diamond/Flower crown holding a sign with 2025 printed on it.”

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

58

Experimental Results on Controllable Video Generation

Shen et al. Controllable Video Generation with Provable Disentanglement. In submission.

• By identifying the latent temporal process and using the minimal change principle, the video generation process can also be controlled.

Ours

StyleGAN-v

Ours

StyleGAN-v

Ours

StyleGAN-v

Camera Motion

Talking Head

Clouds Drift

Three examples with the same control variable

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

59

Summarization

Sufficient Change

Causal Representation Learning

Sparsity Constraint

Transfer Learning

Visual Analysis

Visual Reasoning

Controllable Generation

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

60

Thanks & References

• Yao, Weiran, Guangyi Chen, and Kun Zhang.“Temporally disentangled representation learning.” NeurIPS, 2022.
• Kong, Lingjing, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. “Partial disentanglement for domain adaptation.” ICML, 2022.
• Song, Xiangchen, Weiran Yao, Yewen Fan, Xinshuai Dong, Guangyi Chen, Juan Carlos Niebles, Eric Xing, and Kun Zhang. “Temporally disentangled representation learning under unknown nonstationarity.” NeurIPS, 2023.
• Chen, Guangyi*, Xiao Liu*, Guangrun Wang, Kun Zhang, Philip HS Torr, Xiao-Ping Zhang, and Yansong Tang. “Tem-adapter: Adapting image-text pretraining for video question answer.” ICCV, 2023.
• Li, Zijian, Ruichu Cai, Guangyi Chen, Boyang Sun, Zhifeng Hao, and Kun Zhang. “Subspace identification for multi-source domain adaptation.” NeurIPS, 2023.
• Chen, Guangyi*, Yuke Li*, Xiao Liu, Zijian Li, Eman Al Suradi, Donglai Wei, and Kun Zhang. “LLCP: Learning Latent Causal Processes for Reasoning-based Video Question Answer. ” ICLR 2024.
• Chen, Guangyi*, Yifan Shen*, Zhenhao Chen*, Xiangchen Song, Yuewen Sun, Weiran Yao, Xiao Liu, and Kun Zhang. “CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process.” ICML, 2024.
• Kong, Lingjing*, Guangyi Chen*, Petar Stojanov, Haoxuan Li, Eric Xing, and Kun Zhang. “Towards Understanding Extrapolation: a Causal Lens.” NeurIPS, 2024.

Thanks to the collaborators and our group members: Kun Zhang, Weiran Yao,

Zijian Li, Lingjing Kong, Xiangchen Song, Shaoan Xie, Yifan Shen, …

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

61

Thanks & References

• Li, Yuke*, Guangyi Chen*, Ben Abramowitz, Stefano Anzellott, and Donglai Wei. “Learning Causal Domain-Invariant Temporal Dynamics for Few-Shot Action Recognition.” ICML, 2024.
• Kong, Lingjing, Guangyi Chen, Biwei Huang, Eric Xing, Yuejie Chi, and Kun Zhang. “Learning discrete concepts in latent hierarchical models.” NeurIPS, 2024.
• Song, Xiangchen, Zijian Li, Guangyi Chen, Yujia Zheng, Yewen Fan, Xinshuai Dong, and Kun Zhang. “Causal temporal representation learning with nonstationary sparse transition.” NeurIPS, 2024.
• Li, Zijian, Yifan Shen, Kaitao Zheng, Ruichu Cai, Xiangchen Song, Mingming Gong, Zhengmao Zhu, Guangyi Chen, and Kun Zhang. “On the Identification of Temporally Causal Representation with Instantaneous Dependence.” ICLR, 2025.
• Shaoan Xie, Lingjing Kong, Yujia Zheng, Zeyu Tang, Eric P. Xing, Guangyi Chen, and Kun Zhang. “Aligning Atomic Vision Language Concepts for Controllable Image Generation.” In Submission.
• Yifan Shen, Peiyuan Zhu, Zijian Li, Shaoan Xie, Zeyu Tang, Namrata Deka, Zongfang Liu, Guangyi Chen, and Kun Zhang. “Controllable Video Generation with Provable Disentanglement.” In Submission.
• Ignavier Ng, Yan Li, Zijian Li, Yujia Zheng, Guangyi Chen,  and Kun Zhang. “A General Representation-Based Approach to Multi-Source Domain Adaptation.” in submission.

Thanks for your listening

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

63

Generative Learning Framework

Encoder

Encoder

Encoder

…

…

…

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

64

Summarization

Sufficient Change

Causal Representation Learning

Real-world

Application

Theoretical

Foundation

Sparsity Constraint

Transfer Learning

Visual Analysis

Visual Reasoning

Controllable Generation

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

65

Extension under Label-shift and Complex Transformation

Kong et al. Subspace identification for multi-source domain adaptation. ICML, 2022

Guangyi Chen | Postdoc @ CMU & MBZUAI | chengy12.github.io

66

Extension for a General Case

Kong et al. Partial disentanglement for domain adaptation. ICML, 2022