1 of 37

The BERT Advantage:

Deep Dive into Variants, Multimodal Evolution & Open Source Use Cases

Harini Anand

Data & AI at IBM

she/her

2 of 37

Agenda

● Introduction to BERT

● Overview of Architecture

● BERT Variants Overview

● Open Source Multimodal Use Cases

● Challenges

● Learning Resources

3 of 37

WHY BERT?

4 of 37

source: google images

5 of 37

WHY BERT ?

  • The existing strategies for the pre-trained language representations are mostly based on unidirectional language models and hence are not very effective in capturing the entire context for sentence-level tasks.

​

  • BERT aims to generate deep bidirectional representations by using maked language models, where it is crucial to capture context from both directions.

6 of 37

source: BERT preprint

7 of 37

8 of 37

WHAT IS BERT ?

  • Bidirectional Encoder Representations from Transformers.
  • Developed by Google in 2018
  • It is designed to pre-train deep bidirectional representations from unlabeled text. It performs a joint conditioning on both left and right context in all the layers.

​

9 of 37

10 of 37

HOW DOES BERT WORK?

  • Two Main Steps:
  • Pre-training:
      • Train on unlabeled data
      • Tasks: Masked Language Modeling (MLM), Next Sentence Prediction (NSP)

2. Fine-tuning:

      • Fine-tune with labeled data for downstream tasks
      • Task-specific models sharing the same pre-trained parameters

​

11 of 37

HOW DOES BERT WORK?

source: BERT preprint

12 of 37

GLUE RESULTS

The General Language Understanding Evaluation (GLUE) benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems. GLUE consists of:

  • A benchmark of nine sentence or sentence-pair language understanding tasks built on established existing datasets and selected to cover a diverse range of dataset sizes, text genres, and degrees of difficulty,
  • A diagnostic dataset designed to evaluate and analyze model performance with respect to a wide range of linguistic phenomena found in natural language, and
  • A public leaderboard for tracking performance on the benchmark and a dashboard for visualizing the performance of models on the diagnostic set.

(jee for nlp)

13 of 37

ARCHITECTURE OVERVIEW

  • Input representation: tokens, segment IDs, position embeddings
  • Transformer encoder layers: multi-head self-attention, feed-forward neural networks
  • Pre-training: MLM and NSP tasks

​

14 of 37

ARCHITECTURE OVERVIEW

BERT input representation

15 of 37

BERT VARIANTS OVERVIEW

​

  • Reasons for developing variants:
    • efficiency
    • performance improvements
    • specific applications

​

16 of 37

DISTILBERT

  • Developed by Hugging Face
  • Smaller, faster, and cheaper while retaining 97% of BERT's performance by reducing number of layers
  • To leverage the inductive biases learned by larger models during pre-training, the authors introduce a triple loss combining language modeling, distillation and cosine-distance losses
  • Ideal use cases: resource-constrained environments

​

17 of 37

DISTILBERT

  • A process to create a smaller model (student) that mimics a larger model (teacher)
  • Steps:
    • Train the large model (teacher)
    • Use the predictions of the teacher model to train the smaller model (student)
    • The student model learns to approximate the teacher's behavior

​

18 of 37

19 of 37

RoBERTa

  • Developed by Facebook AI
  • Key differences from BERT:
    • No NSP
    • Dynamic masking
    • Larger mini-batches
    • Longer training

​

20 of 37

RoBERTa

  • Advantages:
    • Improved Performance
      • Outperforms BERT on various NLP benchmarks, including GLUE, RACE, and SQuAD
    • No NSP Task
      • Simplifies the pre-training process

​

21 of 37

22 of 37

RoBERTa

  • Advantages:
    • Dynamic Masking
      • With exposure to variety of masked tokens, helps the model generalize better to different contexts and input variations
    • Larger Mini Batches
      • Allows for better utilization of GPU/TPU resources, leading to more efficient training

​

23 of 37

ALBERT

  • Developed by Google Research
  • Focus on efficiency with parameter reduction techniques:
  • factorized embedding parameterization
  • cross-layer parameter sharing

why? to lower memory consumption &

increase the training speed of BERT.

​

24 of 37

Parameter Reduction Techniques

    • Factorized Embedding Parameterization:
      • Separates the size of the hidden layer from the size of the vocabulary embedding
      • Reduces the number of parameters by factorizing the embedding matrix into two smaller matrices
    • Cross-Layer Parameter Sharing:
      • Reduces the total number of parameters while keeping the depth of the network

​

25 of 37

26 of 37

How BERT Paved the Way for Multimodal Models

27 of 37

From BERT to Multimodal AI

  • BERT revolutionized NLP with contextual embeddings and self-attention.�
  • The transformer architecture behind BERT set the stage for multimodal AI.�
  • Multimodal foundation models now extend this to text, images, audio, video, and more.

28 of 37

What Are Multimodal Foundation Models?

  • Definition: Models that process and integrate multiple types of data (text, images, audio, etc.).�
  • Advantages:�
    • Richer understanding of the world.�
    • Cross-modal learning: Improving one modality with another.�
    • Broader applicability (chatbots, search, healthcare, robotics).�
  • Examples: CLIP, Flamingo, Gemini, LLaVA.

29 of 37

30 of 37

Cross-Modal Transfer Learning

Concept: Learning from one modality improves another.�

Examples:�

  • CLIP (OpenAI) – Connects text and images for better understanding.�
  • Whisper – Uses text knowledge to improve speech recognition.�
  • Video-BERT – Learns video + text representations for action recognition.�

Why It Matters: Enables zero-shot learning and efficient adaptation.

31 of 37

Real-Time Multimodal Processing: Challenges & Solns

Why is real-time processing hard?�

  • High computational cost.�
  • Synchronization across multiple data types.�
  • Storage and retrieval complexities.�

Advancements:�

  • Parallelized transformer inference.�
  • Optimized storage solutions (vector databases).�
  • Hardware acceleration (TPUs, GPUs, custom silicon).�

32 of 37

USE CASES

swiss army knife solution to 11+ of the most common language tasks

33 of 37

USE CASES

  • Text classification: sentiment analysis, spam detection
  • Named Entity Recognition (NER): identifying entities in text
  • Question Answering: systems like SQuAD
  • Sentence pair tasks: semantic textual similarity, natural language inference

34 of 37

The Role of Open Source in Multimodal AI

Open-source projects drive innovation and accessibility.

Notable contributions:

  • Hugging Face Transformers – democratized NLP and multimodal models.�
  • LAION dataset – enabling large-scale multimodal training.�
  • OpenCLIP – open-source alternative to OpenAI’s CLIP.�

Importance of community contributions and ethical considerations.

​

35 of 37

CODE DEMOS

A Visual Notebook for First Time BERT Explorers

source: huggingface tutorials

36 of 37

TAKEAWAY RESOURCES

37 of 37

Q&A

Feel free to reach out to me on LinkedIn and Twitter