O-TPT: Orthogonality Constraints for Calibrating Test-Time Prompt Tuning in Vision-Language Models

GIthub Repository

Motivation

• Vision-language models (VLMs) are powerful but struggle with calibration.

​

• Test-time prompt tuning (TPT) improves accuracy, but often overconfident.

​

• Poor calibration is risky in domains like healthcare and autonomous systems.

​

• Existing methods like C-TPT focus on L2 dispersion but ignore angular structure.

​

Core Insight

​

• Calibration error correlates with cosine similarity between text features

• Greater angular separation = better calibration.

• Dispersion alone (via L2 norm) underutilises feature space.

Our Method – O-TPT

Orthogonality constraint on class text features

• Encourages low cosine similarity between text embeddings

• Promotes better angular utilisation of hyperspherical space

Experimental Setup

• Backbones: CLIP-B/16 (ViT) and CLIP-RN50 (ResNet)

• Tasks:

• Fine-grained: Food101, DTD, Pets, Cars, etc. (9 datasets)

• General: ImageNet, Caltech101

• Natural distribution shifts: ImageNet-A, V2, R, S

• Metrics: Accuracy, Expected Calibration Error (ECE), Static Calibration Error (SCE)

Experiments

CLIP-B/16

CLIP-RN50

Experiments

Natural Distribution Shift

Calibration Quality – Reliability Diagrams

• C-TPT shows drift from diagonal (poor calibration)

• O-TPT stays close to diagonal (well-calibrated)

Conclusion

• O-TPT improves calibration in test-time prompt tuning without requiring supervision

• Enforcing orthogonality on textual features leads to more reliable confidence estimates

• Consistently achieves lower ECE across datasets, backbones, and domain shifts

• Future Direction:

• Dynamic Prompt Regularization: Adjust orthogonality strength based on uncertainty or input shifts.

​

• Medical and Safety-critical Applications: Deploy O-TPT in high-risk domains like radiology or autonomous driving where calibrated confidence is essential.

Thank You

GIthub Repository