Fusing Vision and Language�A Tutorial on Vision-Language Models for Multimodal Content Analysis

Dr.-Ing. Eric Müller-Budack, Sushil Awale

TIB – Leibniz Information Centre for Science and Technology

Research Group Visual Analytics

Slide 1

Agenda

• 13:30 - 13:45 Welcome Session
• 13:45 - 15:30 From Language and Vision to Vision-Language Models
• 15:30 - 16:00 Coffee Break
• 16:00 - 17:30 Generative AI & Video Analysis
• 17:30 - 18:00 Discussion and Closing Session

Slide 2

Research Group Visual Analytics (TIB)

• Since November 2015 at TIB
• Lead: Dr.-Ing. Eric Müller-Budack, Prof. Dr. Ralph Ewerth
• 16 members (thereof 11 PhD students)

​

Research interests:

• Multimodal & cross-modal information retrieval
• Deep learning for the analysis and annotation of media data
• Interactive search, exploration, and visualization of media data
• Open Research Knowledge Graph (https://orkg.org/)
• Digital library as a virtual place of learning

​

Slide 3

Research Group Visual Analytics (TIB)

• Since November 2015 at TIB
• Lead: Dr.-Ing. Eric Müller-Budack, Prof. Dr. Ralph Ewerth
• 16 members (thereof 11 PhD students)

​

Research interests:

• Multimodal & cross-modal information retrieval
• Deep learning for the analysis and annotation of media data
• Interactive search, exploration, and visualization of media data
• Open Research Knowledge Graph (https://orkg.org/)
• Digital library as a virtual place of learning

​

Slide 4

Research Projects (past 5 years)

Text &

Image

Retrieval

Innovative

Services

Digital

Learning

Places

Multimodal

Information

FakeNarratives

ReflectAI

Slide 5

Research Projects (past 5 years)

Text &

Image

Retrieval

Innovative

Services

Digital

Learning

Places

Multimodal

Information

FakeNarratives

ReflectAI

Slide 6

Multimodal Information Extraction

Slide 7

Springstein et al. @ACMMM'21: iART: A Search Engine for Art-Historical Images to Support Research in the Humanities. ACM Multimedia Conference.

Slide 8

Awale et al. @PatentSemTech'21: Exploring Patents Visually: An Interactive Search System for Multimodal Patent Image Search and Interpretation. Workshop on Patent Text Mining and Semantic Technologies co-located at ACM SIGIR’25

Slide 9

Research Projects (past 5 years)

Text &

Image

Retrieval

Innovative

Services

Digital

Learning

Places

Multimodal

Information

FakeNarratives

ReflectAI

Slide 10

Springstein et al. @SIGIR'23: TIB AV-Analytics: A Web-based Platform for Scholarly Video Analysis and Film Studies. International ACM SIGIR Conference on Research and Development in Information Retrieval.

Slide 11

Research Projects (past 5 years)

Text &

Image

Retrieval

Innovative

Services

Digital

Learning

Places

Multimodal

Information

FakeNarratives

ReflectAI

Slide 12

Workshop Resources

tib-visual-analytics.github.io/Konvens2025-FusingLanguageWithVision/

https://www.menti.com/alrqar17xmoc

Slide 13

Agenda

• 13:30 - 13:45 Welcome Session
• 13:45 - 15:30 From Language and Vision to Vision-Language Models
• 15:30 - 16:00 Coffee Break
• 16:00 - 17:30 Generative AI & Video Analysis
• 17:30 - 18:00 Discussion and Closing Session

Slide 14

From Language and Vision to Vision-Language Models

Slide 15

Computer Vision

Convolutional NNs (+ResNets)

Natural Lang. Proc.

Recurrent NNs (+LSTMs)

Translation

Seq2Seq

Speech

Deep Belief Nets (+non-DL)

h e l l o

[1] CNN image CC-BY-SA by Aphex34 for Wikipedia https://commons.wikimedia.org/wiki/File:Typical_cnn.png

[2] RNN image CC-BY-SA by GChe for Wikipedia https://commons.wikimedia.org/wiki/File:The_LSTM_Cell.svg

​

GRBM

🔒

RBM

🔒

RBM

DBN

DBN

DBN

h e l l o </s> s a l u t

s a l u t </s>

Reinf. Learning

BC/GAIL

[1]

​

[2]

​

Slide taken from "Transformers" ©Lucas Beyer lbeyer@google.com

Slide 16

Computer Vision

Natural Lang. Proc.

Translation

Speech

Reinf. Learning

Graphs/Science

Transformer image source: "Attention Is All You Need" paper

Slide taken from "Transformers" ©Lucas Beyer lbeyer@google.com

Slide 17

New Opportunities and Applications

Contrastive Language-Image Pretraining

[Radford et al. 2021]

Slide 18

GPT-5

Slide 19

From Language and Vision to Vision-Language Models

​

• Overview Natural Language Processing (NLP)
• Recurrent Neural Network (RNN)
• Long Short-Term Memory (LSTM)
• Attention Mechanism & Transformer
• Overview Computer Vision (CV)
• Datasets & Preprocessing
• Convolutional Neural Network (CNN)
• Vision Transformer (ViT)
• "Fusion" of Text and Images
• Types of Multimodal Fusion Strategies
• Language-supervised Learning (CLIP: Contrastive Language-Image Pretraining)
• Demo Session
• Domain Adaptation

Slide 20

Overview�Natural Language Processing (NLP)

Slide 21

Neural History of NLP (until 2020)

Classic approaches

N-gram language modeling�[Kneser & Ney et al., 1995]

Ruder S., 2018: A Review of the Neural History of Natural Language Processing. Blog Post. Retrieved from: ruder.io/a-review-of-the-recent-history-of-nlp/

​

​

Slide 22

Language Modeling

​

​

​

• Classic approaches are based on n-grams�[Kneser & Ney, 1995]
• First neural language model was a feed-forward�neural network [Bengio et al., 2001]
• Simple task with large practical applications

Language modeling is the task of predicting the next word in a text given the previous words.

A feed-forward neural network language model [Bengio et al., 2001]

Kneser & Ney @ICASSP’95: Improved backing-off form m-gram language modeling. International Conference on Acoustics, Speech, and Signal Processing�Bengio et al. @NeurIPS’01: A Neural Probabilistic Language Model. Annual Conference on Neural Information Processing Systems

Slide 23

Word Embeddings

Continuous bag-of-words and skip-gram architectures [Mikolov et al., 2013a; 2013b]

Relations captured by word2vec [Mikolov et al., 2013a; 2013b]

Mikolov et al. @ICLR’13: Efficient Estimation of Word Representations in Vector Space. International Conference on Learning Representations

Mikolov et al. @NeurIPS’13: Distributed Representations of Words and Phrases and their Compositionality. Annual Conference on Neural Information Processing Systems

Slide 24

Neural networks of NLP

Fig: A long short-term memory network [Hochreiter & Schmidhuber et al., 1997]

Fig: A recurrent neural network [Elman et al., 1990]

Fig: Convolutional neural network for text [Kim et al., 2014]

Fig: A recursive neural network [Socher et al., 2013]

Elman et al. @Cognitive Sciences’90: Finding structure in time. Cognitive Science

Kim et al. @EMNLP’14: Convolutional Neural Networks for Sentence Classification. Empirical Methods in Natural Language Processing

Hochreiter & Schmidhuber @Neural Computation’97: Long short-term memory. Neural computation

Socher et al. @EMNLP’13: Recursive deep models for semantic treebank. Empirical Methods in Natural Language Processing

Slide 25

Sequence-to-sequence models [Sutskever at al. @NeurIPS’14]

Sequence-to-sequence [Sutskever et al., 2014] models

• Encoder compresses input to a vector representation
• Decoder predicts output based on context vector

​

​

​

​

​

​

​

• Context vector was a bottleneck; challenging to deal with long sentences

Sutskever et al. @NeurIPS’14: Sequence to sequence learning with neural networks. Annual Conference on Neural Information Processing Systems

Alammar J. 2018: Visualizing A Neural Machine Translation Model (Mechanics of Seq2Seq Models with Attention. Blog Post. Retrieved from: jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/

​

​

Slide 26

Attention [Bahdanau et al. @ICLR’15]

• Attention highly improved the quality of machine translation
• Allows model to pay attention on relevant parts of the input sequence

​

​

​

​

​

​

​

• Encoder passes all the hidden states to the decoder
• Hidden states are weighted; decoder pays attention to different part of input sequence

​

​

​

​

​

​

Bahdanau et al. @ICLR’15: Neural Machine Translation by Jointly Learning to Align and Translate". International Conference on Learning Representations

Alammar J. 2018: Visualizing A Neural Machine Translation Model (Mechanics of Seq2Seq Models with Attention. Blog Post. Retrieved from: jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/

​

​

Slide 27

Transformers [Vaswani et al. @NeurIPS’21]

• Uses self-attention mechanisms
• Computes relationships among tokens in same text sequence
• Captures dependencies between two tokens regardless of distance
• “The animal didn’t cross the street because it was tired.”

​

Vaswani et al. @NeurIPS’17: Attention Is All You Need. Annual Conference on Neural Information Processing Systems

• Runs multiple self-attention heads in parallel
• Focuses on different parts of the input simultaneously�“The cat sat on the mat.”
• Head 1: focuses on nearby dependencies (“cat” ↔ “sat”).
• Head 2: focuses on long-range dependencies (“The” ↔ “mat”).
• Head 3: captures function words (“on” ↔ “the”).

• Mitigates vanishing gradients problem with Residual Connection
• Scales well with data and model size at the cost of memory

Slide 28

Transformer with Input Sequence

Alammar J. 2018: The Illustrated Transformer. Blog Post. Retrieved from: jalammar.github.io/illustrated-transformer/

​

​

Slide 29

Pretrained language models

Wang & Zhang, 2020: Must-Read Papers on Pre-trained Language Models. GitHub Repo. Retrieved from: github.com/thunlp/PLMpapers

​

​

Slide 30

Decoder-only

GPT

Encoder-only

BERT

Enc-Dec

T5

[START] [The_] [cat_]

[sat_]

Transformer image source: "Attention Is All You Need" paper

Slide taken from "Transformers" ©Lucas Beyer lbeyer@google.com

Slide 31

Overview�Computer Vision (CV)

Slide 32

Progress in Computer Vision since 2010

​

​

• Huge progress in many tasks such as object classification, person recognition, etc.
• Better than humans in some tasks

​

​

Top-5 error for 1,000 classes in the ImageNet Large Scale Visual

Recognition Challenge 2012 (ILSVRC 2012; Russakovsky et al. @IJCV'15)

Krizhevsky et al. @NeurIPS'12

Krizhevsky et al. @NeurIPS'12: ImageNet Classification with Deep Convolutional Neural Networks. Annual Conference on Neural Information Processing Systems.

Russakovsky et al. @IJCV'15: ImageNet Large Scale Visual Recognition Challenge. International Journal on Computer Vision, 115(3).

Slide 33

Can you name the animal(s)?

Two distinct classes of the Visual Object Recognition Challenge

Siberian Husky

Eskimo Dog

Images from Szegedy et al. @CVPR'15: Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition.

Slide 34

Where was this image taken?

https://labs.tib.eu/geoestimation

Müller-Budack et al. @ECCV'18: Geolocation Estimation of Photos Using a Hierarchical Model and Scene Classification. European Conference on Computer Vision.

Slide 35

Screenshot from�https://lukashaas.github.io/PIGEON-CVPR24/

Slide 36

Challenges in Computer Vision

Slide 37

Deep Learning for Image Classification

​

Images are just a matrix (or tensor) of numbers

Slide 38

Deep Learning for Image Classification

​

Images are just a matrix (or tensor) of numbers

We could just linearize the image and use a fully-connected neural network

Slide 39

Drawbacks of Fully-connected Neural Networks

• Entire image is linearized

​

• No locality preservation
• Spatial information get lost initially

​

• All neurons are connected to each other
• Many parameters (each connection = one weight to learn)

​

​

How can we use the spatial structure of the input?

​

​

Slide 40

Visual Cortex

• Investigation how mammals (cats, monkeys) perceive the world (Hubel & Wiesel 1962)
• Visual cortex includes neurons that respond to direct environment

Image from http://neuroclusterbrain.com/neuron_model.html

Slide 41

Feature Extraction with Convolution

Idea: Connect patches to single neurons

​

​

​

​

​

​

​

Filter is convolved with the input to generate a single output

This step is repeated using a sliding window

Filter weights are automatically learned during the training

Input

Kernel

Output�(Feature Map)

Image from https://s.academiccloud.de/QMQynd

Slide 42

Convolutional Layer

• Convolutional filters are:
• spatially shared (reduces memory, parallelizable)
• learned automatically during training
• Each filter produces one feature map
• Non-linear activation (e.g., ReLU) on each feature map

​

​

Image from Müller-Budack @LUH'21: Unsupervised quantification of entity consistency between photos and text in real-world news. Gottfried Wilhelm Leibniz Universität Hannover.

Image from https://m-alcu.github.io/blog/2018/01/13/neural-layers/

Slide 43

Convolutional Neural Network

Image from Zeiler & Fergus @ECCV'14: Visualizing and Understanding Convolutional Networks. European Conference on Computer Vision.

Slide 44

Popular CNN Architectures

• 2012: AlexNet (Krizhevsky et al. @NeurIPS'12) �8 layers, 61 million parameters
• 2014: VGG (Simonyan & Zisserman @ICLR'15)�19 layers
• 2014: GoogLeNet (Szegedy et al. @CVPR'15)�“Inception” modules, 22 layers, 5 million parameters

ImageNet

Results

Krizhevsky et al. @NeurIPS'12: ImageNet Classification with Deep Convolutional Neural Networks. Annual Conference on Neural Information Processing Systems.

Simonyan & Zisserman @ICLR'15: Very Deep Convolutional Networks for Large-Scale Image Recognition. International Conference on Learning Representations.

Szegedy et al. @CVPR'15: Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition.

He et al. @CVPR'16: Deep Residual Learning for Image Recognition. IEEE Conference on Computer Vision and Pattern Recognition.

• 2015: ResNet (He et al. @CVPR'16)�Residual layers, 152 layers

Slide 45

Residual Layer (He et al. @CVPR'16)

• Vanishing gradient problem with very deep networks
• Backpropagation leads to small gradients in the first layers
• Gradient commonly lower than 1
• Chain rule ➜ Multiplication of derivations (gradient ➜ 0)
• Performance of very deep networks decreases
• Residual layer with skip connection
• z = x + g(y)
• Local gradient of identity function ∂x/∂z = 1
• Preserves gradient ➜ allows deeper networks

​

​

x

g(y)

z

x

He et al. @CVPR'16: Deep Residual Learning for Image Recognition. IEEE Conference on Computer Vision and Pattern Recognition.

Slide 46

ResNet (He et al. @CVPR'16)

Increase of the maximum number of layers from 19 to over 152

He et al. @CVPR'16: Deep Residual Learning for Image Recognition. IEEE Conference on Computer Vision and Pattern Recognition.

Slide 47

Residual Connection in Transformers

Slide 48

From Transformer to Vision Transformer (ViT)

​

​

​

​

​

​

​

​

Images are continuous

How to use them as transformer input?

​

​

​

​

​

​

​

​

Language is sequential

Text tokens can be directly used as input

​

​

Slide 49

From Transformer to Vision Transformer (ViT)

Image patches as input sequence

Self-attention at the pixel level?

For 224px² ➜ 50.176 tokens

Too much!

"tokenize" the image with image patches

For patches of size 16px² ➜ 196 tokens

Slide 50

From Transformer to Vision Transformer (ViT)

Dosovitskiy et al. @ICLR'21: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. International Conference on Learning Representations.

Slide 51

Attention in Vision Transformers

Slide 52

Attention in Vision Transformers

Image from Dosovitskiy et al. @ICLR'21: An Image is Worth 16x16 Words:

Transformers for Image Recognition at Scale. International Conference on

Learning Representations.

Transformers can pay attention to distant regions even in early layers

Image taken from https://introtodeeplearning.com/

Slide 53

Receptive Field of CNNs

Transformers can pay attention to distant regions even in early layers

CNNs attention is limited to the receptive field by design (increases with depth)

Image taken from https://introtodeeplearning.com/

Slide 54

Receptive Field of CNNs

Image from Dosovitskiy et al. @ICLR'21: An Image is Worth 16x16 Words:

Transformers for Image Recognition at Scale. International Conference on

Learning Representations.

CNNs

Theoretical line!�(depends on kernel�size, stride, etc.)

Transformers can pay attention to distant regions even in early layers

CNNs attention is limited to the receptive field by design (increases with depth)

Image taken from https://introtodeeplearning.com/

Slide 55

CNN vs. Vision Transformer (ViT)

Are CNNs obsolete?

Require less training data

Faster, lower computational resources

Less explainable

More powerful with sufficient training data

Require more computational resources

More explainable

It depends on the task and available resources!

Slide 56

Object Detection (Simplified)

Extraction of region proposals (random, region proposal networks, etc.)

Remainder of the pipeline is similar to object classification with an additional background class

Main challenge: efficient ways to propose regions of interest

Image from Girshick et al. @CVPR'14: Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. IEEE Conference on Computer Vision and Pattern Recognition,

Slide 57

Image Segmentation (Simplified)

Encoder-decoder architecture with transpose convolution layers in the decoder

Pixel-wise prediction of the class (including background)

Main challenge: preserve spatial resolution (skip connection, atrous convolution)

Image from https://sthalles.github.io/deep_segmentation_network/

Slide 58

Object Detection & Image Segmentation with Transformers

Example: DEtection TRansformer (DETR) from Carion et al. @ECCV'20

Encoder-decoder architecture

Decoder produces a fix number (e.g., 100) of regions (bounding box) with class probabilities

Carion, et al. @ECCV'20: End-to-End Object Detection with Transformers. European Conference on Computer Vision.

Slide 59

Object Detection & Image Segmentation with Transformers

Example: DEtection TRansformer (DETR) from Carion et al. @ECCV'20

Encoder-decoder architecture

Decoder produces a fix number (e.g., 100) of regions (bounding box) with class probabilities

Can be extended to segmentation by adding a mask head on top of the decoder outputs.

Carion, et al. @ECCV'20: End-to-End Object Detection with Transformers. European Conference on Computer Vision.

Slide 60

"Fusion" of Text and Images

Slide 61

Text vs. Images

How are text and images different?

Heterogenous�Modalities

(with diverse qualities)

Homogenous�Modalities

(with similar qualities)

Images from 2 cameras

Text and Image

A dog catching a frisbee in a park

Image: unsplash.com/photos/black-and-brown-dog-catching-yellow-flying-disc-gcioe8naQys

Slide 62

Multimodal Machine Learning (MML)

Multimodal Machine Learning deals with heterogeneous and interconnected data

MML Tasks

1) Representation: learning representations that capture cross-model interactions

• Fusion: learning a joint representation
• Coordination: learning representations that are coordinated across modalities

2) Alignment: identifying cross-modal connections

3) Generation: produce raw modalities that reflect cross-modal interactions

Other Tasks:

Reasoning, Transference, Quantification

Multimodal Tutorial: cmu-multicomp-lab.github.io/mmml-tutorial/icml2023/

Slide 63

Fusion

• Captures rich, joint cross-modal representations by combining features at input stage
• Simpler training pipeline; single model
• Sensitive noise and requires well-aligned modalities
• High dimensionality and risk of overfitting

• Balances modality-specific processing and joint learning
• Preserves important features from each modality
• Architectural complexity
• Optimal fusion is hard; requires extensive tuning

• Robust to missing or asynchronous modalities
• Flexible: easy to add or remove modalities
• May miss rich cross-modal relationships
• Aggregation method selection can affect overall performance

Multimodal fusion strategies [Huang et al. 2020]

Huang et al. @NPJ Digital Medicine’20: Fusion of Medical Imaging and Electronic Health Records Using Deep Learning: A Systematic Review and Implementation Guidelines. NPJ Digital Medicine

Slide 64

Coordination

1. Specialized encoders capture heterogeneity
2. Coordination function captures interactions�e.g., cosine similarity

Coordination with Contrastive Learning

• Contrastive Loss:
• brings positive pairs closer
• pushes negative pairs apart

​

Coordinated multimodal representations�[Baltrušaitis, Ahuja, and Morency, 2017]

Baltrušaitis, Ahuja, and Morency @arXiv’17: Multimodal Machine Learning: A Survey and Taxonomy. arXiv

Slide 65

Image-to-Image Contrastive Learning

• Learn visual representations such that
• Maximize similarity between two views�of a single image (augmentation)
• Minimize similarity between that�image and other images from the batch
• How does this work for multimodal data?

Image-to-image contrastive learning with data augmentation [Chen et al. 2020]

Chen et al., PMLR’20: A simple framework for contrastive learning of visual representations. Proceedings of Machine Learning Research

Slide 66

Language-supervised Pre-training �(Radford et al. @ICML'21, OpenAI)

• Pre-training models in Computer Vision (CV) often use crowd-labeled datasets
• acquisition often such datasets time-consuming
• Crowdsourcing limits dataset size
• Can scalable pre-training methods which learn directly from web text be used for CV?
• Crawled a dataset comprising 400 million image-text pairs from the web
• Language-supervised training to project text and image embeddings in the same space

​

​

Radford et al. @ICML'21: Learning Transferable Visual Models From Natural Language Supervision. International Conference on Machine Learning.

Image taken from

​

Schuhmann et al. @NeurIPS-W'21: LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs.

Data Centric AI NeurIPS Workshop 2021.

Slide 67

CLIP – Contrastive Language-Image Pretraining (Radford et al. @ICML'21, OpenAI)

400 million

image-text pairs

(no labeling required)

Example: ResNet, ViT

Example: Transformer

Radford et al. @ICML'21: Learning Transferable Visual Models From Natural Language Supervision. International Conference on Machine Learning.

Slide 68

CLIP Applications

• CLIP learns semantic relations between textual descriptions and images
• This allows for a wide range of applications (even in zero-shot settings):
• Classifications of arbitrary concepts based on natural language descriptions

​

Radford et al. @ICML'21: Learning Transferable Visual Models From Natural Language Supervision. �International Conference on Machine Learning.

Slide 69

CLIP Applications

• CLIP learns semantic relations between textual descriptions and images
• This allows for a wide range of applications (even in zero-shot settings):
• Classifications of arbitrary concepts based on natural language descriptions

​

Robustness of CLIP to distribution shifts [Radford et al., @ICML'21]

Slide 70

CLIP Applications

• CLIP learns semantic relations between textual descriptions and images
• This allows for a wide range of applications (even in zero-shot settings):
• Classifications of arbitrary concepts based on natural language descriptions
• Information retrieval
• Image retrieval (image to image)
• Cross-modal retrieval
• Multimodal retrieval

​

Schuhmann et al. @NeurIPS-W'21: LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs. Data Centric AI NeurIPS Workshop 2021.

Slide 71

CLIP Applications

• CLIP learns semantic relations between textual descriptions and images
• This allows for a wide range of applications (even in zero-shot settings):
• Classifications of arbitrary concepts based on natural language descriptions
• Information retrieval
• Image retrieval (image to image)
• Cross-modal retrieval
• Multimodal retrieval
• Prompt engineering is crucial for the performance
• Standard: Various generic templates such as “a photo of a [class]”
• Can be ambiguous and can lack task-specific context

​

Slide 72

Prompt Engineering for CLIP

Improvements can be made by:

• careful prompt engineering, e.g., with external knowledge [Tahmasebzadeh et al. @WACV'24]
• prompt learning, e.g., Context Optimization [Zhou et al. @IJCV'22]

Tahmasebzadeh et al. @WACV'24: Few-Shot Event Classification in Images using Knowledge Graphs for Prompting. IEEE/CVF Winter Conference on Applications of Computer Vision 2024.

Zhou et al. @IJCV'22: Learning to Prompt for Vision-Language Models. International Journal of Computer Vision, 130(9).

Slide 73

CLIP Classification Demo

Slide 74

Tasks for the Image Classification Demo

1. What happens if the image only show concepts that are not in the dictionary?
2. What happens if multiple concepts in your dictionary are visible in the image?
3. Experiment with negative prompts alongside positive prompts

e.g. “not a photo of a cat”, “photo of a dog”

• Experiment with synonymous prompts

e.g. “a photo of a cat”, “a photo of a kitty”

• Experiment with prompts with visual attributes

e.g. “a red car”, “a black car”

• Use a photo with text in it and see whether CLIP can “read”

​

​

​

​

​

Slide 75

CLIP Retrieval Pipeline

Schuhmann et al. @NeurIPS-W'21: LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs. Data Centric AI NeurIPS Workshop 2021.

Index

Indexing

Text Query

Image Query

User Interface

Results

Slide 76

CLIP-based Image Retrieval Services

iart.vision

service.tib.eu/ipatent

Slide 77

CLIP Retrieval Demo

Slide 78

Tasks for the Image Retrieval Demo

• Retrieve images based on 3-5 query images
1. Are the results meaningful?
2. What kind of similarity do you think CLIP is measuring?
• Retrieve images using ~5 simple prompts (This is a photo of <class>)
• How would you rate the results?
• Are the results intuitive and explainable?
• Try to formulate more advanced prompts, e.g., by adding a short description of the concept
• Do the results improve?
• Can these act as a filter for the search results?
• What can we do to improve search results and explainability?

​

Slide 79

What’s CLIP?

Contrastive Language-Image Pretraining

[Radford et al. 2021]

400 million

image-text pairs

(no labeling required)

Good News

CLIP allows for image retrieval, cross-modal retrieval, and

zero-shot concept recognition for arbitrary concepts!

Bad News

CLIP is not optimized for specialized domains!

Slide 80

CLIP Limitations

• Poor generalization to images not seen during training
• CLIP’s zero-shot classifiers can be sensitive to wording�or phrasing
• How can we adapt CLIP to a target domain?

Slide 81

Finetuning CLIP

Linear Head

Image

Encoder

Label

Linear Head

Image

Encoder

Label

Image

Encoder

Linear Probing

Full Finetuning

Contrastive Pretraining

Text

Encoder

A photo of a [label]

Using cross-entropy loss

Using contrastive loss

Frozen

Finetuned

Goyal et al. @CVPR’23: Finetune like you pretrain: Improved finetuning of zero-shot vision models. IEE Conference on Computer Vision and Pattern Recognition.

Slide 82

Finetuning CLIP

​

Image

Encoder

Text

Encoder

A photo of a [label]

Locked-image Tuning

Image

Encoder

Text

Encoder

A photo of a [label]

Image

Adapter

Text

Adapter

CLIP-Adapter

Contrastive Loss

A photo of a [label]

Contrastive

Learning

Contrastive

Learning

Zhai et al. @CVPR’22: LiT : Zero-Shot Transfer with Locked-image text Tuning. IEEE Conference on Computer Vision and Pattern Recognition.

Gao et al. @IJCV’22: CLIP-Adapter: Better Vision-Language Models with Feature Adapters. International Journal of Computer Vision.

Yang et al. @CVPR’22: CLIP-KD: An Empirical Study of CLIP Model Distillation. IEEE Conference on Computer Vision and Pattern Recognition.

Knowledge Distillation

KL-divergence Loss

Slide 83

Low-Rank Adaptation [Hu et al. @ICLR’22]

Hu et al. @ICLR’22: LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations

Images from ibm.com/think/topics/lora

Slide 84

Guided Demo on Finetuning CLIP

Slide 85

Linear Probing of CLIP for Geolocation Estimation

Only 1 Mio images!

Slide 86

Agenda

• 13:30 - 13:45 Welcome Session
• 13:45 - 15:30 From Language and Vision to Vision-Language Models
• 15:30 - 16:00 Coffee Break
• 16:00 - 17:30 Generative AI & Video Analysis
• 17:30 - 18:00 Discussion and Closing Session

Slide 87

Generative AI

Slide 88

Large Language Models (LLMs)

Naveed et al. @ACMTIST’25. A Comprehensive Overview of Large Language Models. ACM Transactions on Intelligent Systems and Technology

Slide 89

Beyond Language: Rise of Multimodal LLMs

• LLMs solely work with text
• Multimodal LLMs (MLLMs) extend this capability to images, audio and video
• Perform tasks that are not possible with text alone, e.g.
• Describe a photo
• Answer questions about a video clip
• Create a website from a hand-drawn sketch
• Main Challenge: Teach a model to understand concepts in a unified way, e.g.�Sequence of letters ‘d-o-g’ == pixels of a furry, four-legged animal

Zhang et al. @arXiv’24: MM-LLMs: Recent Advances in MultiModal Large Language Models. arXiv

Slide 90

Growing number of MLLMs

Zhang et al. @arXiv’24: MM-LLMs: Recent Advances in MultiModal Large Language Models. arXiv

Slide 91

Key Components of MLLMs’ Architecture

Modality

Encoder

​

​

Input

Projector

LLM

Backbone

Output

Projector

Modality

Generator

Raw data

(images, video)

Embeddings

Embeddings

Aligned Features

Aligned Features

Processed information

Processed information

Modality

Features

Text

Modality

Features

Output

(image, video)

Zhang et al. @arXiv’24: MM-LLMs: Recent Advances in MultiModal Large Language Models. arXiv

Slide 92

Key Components of MLLMs’ Architecture

General model architecture of MLLMs and the implementation choices for each component [Zhang et al., 2024]

Zhang et al. @arXiv’24: MM-LLMs: Recent Advances in MultiModal Large Language Models. arXiv

Slide 93

Two Stage Training Paradigm

Phase 1: Pre-training (PT)

​

Goal: Feature Alignment

​

Train Input Projector to translate visual features into a language LLM can understand

​

Component Trained: Input Projector (e.g. MLP)

​

Data Used: Large-scale image-text pairs

​

Output: Visual concepts are aligned with the LLM’s word embedding space

Phase 2: Instruction Turning (IT)

​

Goal: End-to-End Fine-tuning

​

Train the model to follow instructions with multimodal input and perform reasoning

​

Component Trained: Input Projector and LLM

​

Data Used: High-quality multimodal

instruction-following datasets

​

Output: MLLM that can perform complex Visual Question Answering and dialogue

​

Zhang et al. @arXiv’24: MM-LLMs: Recent Advances in MultiModal Large Language Models. arXiv

Slide 94

Approaches for Modality Alignment [Zhang et al., @arXiv’24]

Vision-language Connectors

• Take powerful, pre-trained, frozen unimodal models (e.g. a vision encoder and an LLM)
• Connect them with a small, lightweight, trainable ‘bridge’
• This bridge is the Input Projector, typically a small MLP
• Parameter-efficient but quality capped by quality of unimodal models; projector can be a bottleneck
• E.g. LLaVA, BLIP-2

Cross-attention Fusion

• Inject visual information into multiple layers of LLM using cross-attention mechanisms
• LLM can refer to visual features at various stages of its reasoning process
• Richer, more fine-grained interplay between modalities
• Complex architecture and computationally expensive
• E.g. Flamingo, OpenFlamingo

End-to-End Training

• A single unified transformer model trained from ground up on massive multimodal corpus
• No separate ‘unimodal’ components
• Powerful models; no constraints by pre-trained components
• Huge training costs; not open-source
• E.g. GPT-4V, Google Gemini

Zhang et al. @arXiv’24: MM-LLMs: Recent Advances in MultiModal Large Language Models. arXiv

Slide 95

BLIP-2 [Li et al. @PMLR’23]

Li et al. PMLR’23: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. Proceedings of Machine Learning Research

Slide 96

LLaVA [Liu et al. @NeurIPS’23]

Liu et al., NeurIPS’24: Visual Instruction Tuning". Annual Conference on Neural Information Processing Systems

Slide 97

Flamingo [Alayrac et al. @NeurIPS’22]

Alayrac et al. @NeurIPS’22: Flamingo: a Visual Language Model for Few-Shot Learning. Annual Conference on Neural Information Processing Systems

Slide 98

Use cases of Large Vision-Language Models (LVLMs)

LVLMs can be used for a wide range of tasks

• visual concept classification
• optical character recognition (OCR)
• image captioning
• etc.

Several benchmarks to evaluate LVLMs exist

• MMBench (Liu et al. @ECCV'24)
• SeedBench (Li et al. @CVPR'24)
• MMStar (Chen et al. @NeurIPS'24)
• etc.

Liu et al. @ECCV'24: MMBench: Is Your Multi-modal Model an All-Around Player? European Conference on Computer Vision.

Li et al. @CVPR'24: SEED-Bench: Benchmarking Multimodal Large Language Models. IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Chen et al. @NeurIPS'24: Are We on the Right Way for Evaluating Large Vision-Language Models? Annual Conference on Neural Information Processing Systems.

Slide 99

Use cases of Large Vision-Language Models (LVLMs)

LVLMs can be used for a wide range of tasks

• visual concept classification
• optical character recognition (OCR)
• image captioning
• etc.

Several benchmarks to evaluate LVLMs exist

• MMBench (Liu et al. @ECCV'24)
• SeedBench (Li et al. @CVPR'24)
• MMStar (Chen et al. @NeurIPS'24)
• etc.

Liu et al. @ECCV'24: MMBench: Is Your Multi-modal Model an All-Around Player? European Conference on Computer Vision.

Li et al. @CVPR'24: SEED-Bench: Benchmarking Multimodal Large Language Models. IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Chen et al. @NeurIPS'24: Are We on the Right Way for Evaluating Large Vision-Language Models? Annual Conference on Neural Information Processing Systems.

Slide 100

Prompting LVLMs

As for LLMs, there exist many ways how to prompt LVLMs efficiently

Types of questions

• Open-ended questions: Which objects are shown in this image?
• Multiple-choice questions: When was this photo taken? (a) During the day (b) At night
• Binary-choice question: Was this photo taken indoors? Answer with yes or no!

Prompt style

• Chain-of-thought (CoT)
• RISEN (Role, Instructions, Steps, End goal, Narrowing)
• …

Provided context, e.g., in-context learning

​

​

Slide 101

Limitations of LVLMs

1. Hallucination and Faithfulness
• prioritize linguistic fluency over factual grounding, leading to hallucinated details not present in the image
• generated text may sound plausible but is not visually grounded
2. Computationally Complexity
• require large-scale GPUs/TPUs and significant memory for training and inference
• less feasible for efficiency-critical tasks such as large-scale retrieval
• hybrid pipelines are common: generative LVLMs are combined with non-generative LVLMs (e.g., CLIP) for re-ranking and retrieval efficiency
3. Domain Gap: Performance can drastically degrade in out-of-context tasks

Some limitations can be resolved with parameter-efficient finetuning and instruction tuning

Slide 102

Demo on LVLMs

• Pick several tasks from the MMBench benchmark dataset and assess the quality of results!
• Compare multiple LVLMs such as InstructBLIP, Qwen2.5-VL, and InternVL!
• Compare the results for various prompt types!
• What are the advantages and disadvantages of the prompt types?
• Try multiple choice prompts but change the order of the options. What do you notice?
• Try to extract overlaid text from images using LVLMs!
• How would you rate the quality of the output?
• Can you extract equations from lecture videos?
• Try using the overlaid text as additional context for task 1!
• Try to extract structured information from data visualizations (e.g., bar charts) with LVLMs

https://colab.research.google.com/drive/1SuUxHVAewvT-rMJr-dz5LnlEFCDxArV6

Slide 103

From Images to Videos

Slide 104

Video Data

• Videos consist of a sequence of frames

​

​

​

​

​

​

​

​

• Approaches for image classification can be applied on each frame
• Temporal context and audio can improve various tasks (e.g., action recognition)

Image from https://learnopencv.com/introduction-to-video-classification-and-human-activity-recognition/

Slide 105

Overview

• Development from single-frame to multi-frame methods
• Handcrafted features, NN (CNN, RNN), self-supervised (Transformer), combined with LLMs

​

Image from Tang et al. @IEEE TCSVT’25: Video Understanding with Large Language Models: A Survey

​

Slide 106

Temporal Pooling

• Extraction of image features from individual frames (pretrained image classifiers)
• Temporal pooling:
• Simple pooling of the features (e.g., mean, max)
• Sequential model (e.g., LSTM, Optical Flow) to capture temporal dynamics
• Classification layer to predict the target class

Slide 107

3D CNN

T video frames are stacked as input tensor: width x height x (3 x T)

Remaining architecture like in CNNs for image classification

Problem: Very large input size depending on the video length (i.e., number of frames T)

​

Slide 108

Video Preprocessing and Tokenization in Transformers

Slide 109

X-CLIP

Slide 110

VideoMAE

Self-supervised pre-training of a video transformer

During training, masked frames (90% - 95%) are recreated in an encoder-decoder architecture

Learns rich semantic video features

During inference, a classification layer is used on top of the encoder for downstream tasks

Slide 111

Video-based LVLM (Video-LLaMA)

• Extension of pretrained LLMs for video analysis
• Visual encoder (e.g. ViT) to extract visual features for each frame in a video sequence
• Audio encoder to extract audio features
• Trainable adapter (e.g., QFormer) to convert the features into the token space of the LLM

​

Problems:

• Computationally expensive
• Less training data available compared to images

Zhang et al. @EMNLP’23: Video-LLaMA An Instruction-tuned Audio-Visual Language Model for Video Understanding

Slide 112

Springstein et al. @SIGIR'23: TIB AV-Analytics: A Web-based Platform for Scholarly Video Analysis and Film Studies. International ACM SIGIR Conference on Research and Development in Information Retrieval.

Slide 113

Agenda

• 13:30 - 13:45 Welcome Session
• 13:45 - 15:30 From Language and Vision to Vision-Language Models
• 15:30 - 16:00 Coffee Break
• 16:00 - 17:30 Generative AI and Domain Adaptation
• 17:30 - 18:00 Discussion and Closing Session

Slide 114

Thank you for your attention!

Slide 115

Ollama Colab

Using colabxterm to start a console in colab runtime

• %xterm # This will start a console
• curl https://ollama.ai/install.sh | sh # This will download and install colab
• ollama serve & # This will start ollama in a background process
• You don’t have to wait for the prompt to show up again here
• ollama pull qwen2.5vl:3b # This will download qwen2.5vl to the ollama cache
• After that you can use the python API to work with ollama

​

​

​

​

Slide 116

Slide 117

Slide 118

Slide 119

Slide 120