1 of 25

Heatmap-Guided Spatial Weighting Improves Class-Discriminative SAE Feature Vector Extraction in Vision Transformers

Master’s Final Defense

2 of 25

Table of Contents

Introduction
Background & Related Work
Methodology
Experiments & Results
Conclusion

Chavakan YIMMARK | Master’s Final Defense

2

3 of 25

What is Vision Transformer

Based on Transformer architecture
Example usages:

Medical image analysis
Autonomous driving
Part of Vison-Language Model (VLM)

Good performance 🡪 Complex model 🡪 Hard to interpret

Chavakan YIMMARK | Master’s Final Defense

3

4 of 25

Explainable AI

Chavakan YIMMARK | Master’s Final Defense

4

Mechanistic Interpretability

This leads to a question of : Can we explain these model?�The answer is: We can do this to some extent. Explainable AI methods help us understand the model decision more.

�First, let's talk about mechanistic interpretability, or mech interp for short.

“Mech interp helps us understand what is happening inside the model and why it makes certain decisions by looking at its internal information

For example, we can use a Sparse Autoencoder, or SAE, to extract features from the. Each feature can correspond to some concept that the model has learned.

We can then look at the examples that strongly activate a particular feature and try to understand what that feature represents.

In this example, feature F2 is activated by images containing aircraft, so we can interpret F2 as an ‘aircraft’ feature.”

5 of 25

PatchSAE

Chavakan YIMMARK | Master’s Final Defense

5

PatchSAE [4] applied Sparse Autoencoder (SAE) to extract feature vectors from patch tokens in ViT

6 of 25

PatchSAE’s Problem

Chavakan YIMMARK | Master’s Final Defense

6

Problem

PatchSAE provides patch-level SAE features, but not an image-level vector
Averaging all patches can cause background features to dilute object features
How can we identify which patches are important?

7 of 25

Explainable AI

Chavakan YIMMARK | Master’s Final Defense

7

Mechanistic Interpretability

Post-hoc Interpretability

8 of 25

Post-hoc Interpretability

Chavakan YIMMARK | Master’s Final Defense

8

Many post-hoc methods exist
Different post-hoc methods can produce different heatmaps

RQ1: Can post-hoc methods help with dilution?

RQ2: How do we choose a good post-hoc method?

9 of 25

HG-PatchSAE (Our framework)

Chavakan YIMMARK | Master’s Final Defense

9

10 of 25

Experimental Setup

Chavakan YIMMARK | Master’s Final Defense

10

Evaluation Metric

Top-K Feature Faithfulness (FF)
Energy-based Pointing Game (EPG) [7]

Configuration

Model

CLIP ViT-B/16 [5]

Dataset

ImageNet-1k [6]�

Post-hoc methods

Raw Attention, Attention Rollout, LeGrad, Grad-SAM, �Transformer Attribution, LayerCAM�

11 of 25

Top-K Feature Faithfulness (FF)

Chavakan YIMMARK | Master’s Final Defense

11

Top-K On: Keep the most active features Top-K Off: Remove the most active features

Top-K Feature Faithfulness (FF): How faithfully the Top-K features capture the information used by the model

We first look at the Top-K Feature Faithfulnesshe, or FF. The goal is to measures how important the top-k features in the extracted vector are to the model prediction.

First, we create a class prototype by averaging the image-level feature vectors from images belonging to the same class as the image on the left.

Then, for each image in the test set, we perform a causal intervention. Changing the model internal information using SAE. Because SAE can extract the feature from the internal information and turn it back be can change the feature vector and put back to the model to change its behavior.

In Top-K On, we retain the Top-K most activating features according to the class prototype while Top-K Off, we remove those Top-K features.

In this example, Top-K On keeps the most activating features, and we get the original prediction, Airplane. But top-k off remove the features related to airplane so the model predict car instead.

We combine the Top-K On and Top-K Off results into a single score, which we call Top-K Feature Faithfulness, or FF. Intuitively, the larger the difference between the two, the more important the Top-K features are to the model's prediction.

12 of 25

Results (FF)

Chavakan YIMMARK | Master’s Final Defense

12

13 of 25

Results (FF)

Chavakan YIMMARK | Master’s Final Defense

13

14 of 25

Results (FF)

Chavakan YIMMARK | Master’s Final Defense

14

15 of 25

Energy-based pointing Game (EPG)

Chavakan YIMMARK | Master’s Final Defense

15

EPG measures how much heatmap mass falls inside the object
Higher EPG → better object-focused heatmap
EPG can be used to select heatmaps for our framework

16 of 25

Results (EPG)

Chavakan YIMMARK | Master’s Final Defense

16

17 of 25

Conclusion

Chavakan YIMMARK | Master’s Final Defense

17

Findings

Post-hoc heatmaps can be used as pooling weights
EPG score can quantify heatmap quality for our framework

Limitation

Evaluated on one backbone and one dataset

Future work

Develop post-hoc methods specifically for feature extraction

18 of 25

References

Chavakan YIMMARK | Master’s Final Defense

18

[1] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, International Conference on Learning Representations (ICLR) (2021)

[2] N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah. Toy Models of Superposition, Transformer Circuits Thread (2022)

[3] A. Templeton, T. Conerly, J. Marcus, J. Lindsey, T. Bricken, B. Chen, A. Pearce, C. Citro, E. Ameisen, A. Jones, H. Cunningham, N. L. Turner, C. McDougall, M. MacDiarmid, A. Tamkin, E. Durmus, T. Hume, F. Mosconi, C. D. Freeman, T. R. Sumers, E. Rees, J. Batson, A. Jermyn, S. Carter, C. Olah, and T. Henighan. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread (2024).

[4] H. Lim, J. Choi, J. Choo, and S. Schneider. Sparse Autoencoders Reveal Selective Remapping of Visual Concepts During Adaptation, International Conference on Learning Representations (ICLR), pp.46012–46037 (2025)

[5] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning Transferable Visual Models From Natural Language Supervision, International Conference on Machine Learning (ICML), pp.8748–8763 (2021)

[6] J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.248–255 (2009)

[7] H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding, P. Mardziel, and X. Hu. Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks, IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp.24–25 (2020)

19 of 25

Appendix: Results (NCC)

Chavakan YIMMARK | Master’s Final Defense

19

20 of 25

Appendix: Results (NCC)

Chavakan YIMMARK | Master’s Final Defense

20

21 of 25

Appendix: Results (NCC)

Chavakan YIMMARK | Master’s Final Defense

21

22 of 25

Appendix: Results (FF with ON/OFF)

Chavakan YIMMARK | Master’s Final Defense

22

23 of 25

Appendix: Class Prototype Construction and Data Subset

Chavakan YIMMARK | Master’s Final Defense

23

For each class in the dataset, we create class prototype vector from 10% of the training set with equation:

Data subset:

Top-K Feature Faithfulness: class prototypes are built from 130 randomly drawn training images per class (10% of training set) and evaluates on ImageNet validation split of 50,000 images
Nearest Centroid Classification: reuses the same 130-image-per-class prototypes and test on the full validation split of 50,000 images

24 of 25

Appendix: Mechanistic Interpretability

Chavakan YIMMARK | Master’s Final Defense

24

Polysemanticity [2] makes it hard to interpret concept(s) from a single neuron

Sparse Autoencoder (SAE) turns the dense, hard to interpret vector into a sparse, more interpretable one [3]

25 of 25

Appendix: CLIP

Chavakan YIMMARK | Master’s Final Defense

25