Lips Sometimes Lie: Towards Understanding Lip Reading Networks

​

Submitted By

Sree Harsha Nelaturu, 7023514

Ashwath Shetty, 7025887

Neel Kelkar, 7024010

High Level Computer Vision Project Summer Semester 2022

Link to Slides

https://docs.google.com/presentation/d/1RjF42bKE_cqWNywJuel9ZkycdB3lbPi276WROWEYIAE/edit?usp=sharing

Lip2Wav

VSPR

Conclusion: Lip-reading is hard

Prediction

​

rst question is to fresh a high level of ability to catch expertise a couple of fields martial arts ai music and sarah i'm here

​

Prediction

Visual -> Text (without audio)

In short, Multimodal

Visual -> Audio (without text)

Visual + Audio-> Text

Question: Why is Lip Reading so hard?

Source

Question: What could address the challenges?

• Multimodal: Common architecture for different domains.
• Labelled Data: Self-supervised pre-training, fine-tuning capability
• Domain bias: Modality Dropout
• Interpretable Features: Attention
• Best Downstream Viability: Visual -> Text

Where no architecture has gone before: AVHubert

• Self-supervised pre-training via masked cluster prediction.

Source

What’s in it for us? Fine-tuning

• Can be used as an encoder to fine-tune for Lip-reading even in single modality (video -> text).

Source

Question

​

Does this approach fit our requirements?

Is the network learning the right features? We find out

Experiment: We mask different sections of the video to check which features affect the Word Error Rate

Can we extend this further? Flipping the tables

Experiment: The model is able to predict an instance of a word given visual input. We use this capability in reverse, to query frame(s) using a word.

Text: then

Attention Map Token: 85-90

Frames

Important question

​

How robust is the network?

How robust is the network to perturbations? ADVERSARIAL ATTACKS!

Experiment: Given a network, we perturb the input such that it makes a false prediction creating what are referred to as ‘adversarial’ examples.

Source

Generating Adversarial Examples: FGSM

Source

Important question

​

Do these attacks work on AVHubert?

Experiment: Untargeted Adversarial Examples

Prediction

which is that it sounds kind of cool and then like a form like a melody around in and that's a lot of fun like i would

Prediction

i said hey aren't it really occurred it's like yelling out of football and martin luther king and yelling al qaeda and i'd like to site on the other philosophy and you

Experiment: Untargeted Adversarial Examples

​

Prediction

one day a young boy comes upon the sunflower while visiting the garden and he notices how weak it looks

Prediction

so we think about it when the audience are going like the creature of the ground

​

Can we get it to say what we want?

Idea

Input

Model

Loss

Gen Adversarial Examples

Clip to Epsilon Norm

• Naive : Do FGSM Attack to minimize loss to target. Doesn’t Work
• Solution: Iterative Attack

Experiment: RickRolling AVHubert

Prediction

which is that it sounds kind of cool and then like a form like a melody around in and that's a lot of fun like i would

Prediction

never gonna give you up, never gonna let you down

​

Experiment: RickRolling AVHubert

Question

​

How can it be made robust?

Know your enemy: Adversarial Training

Source

Experiment: WER vs Epsilon Results

Experiment: Train / Val Curves

Training and Validation Accuracy for Standard Training

Pink: Train | 72.92

Green: Valid | 69.22

Training and Validation Accuracy for Adversarial Training with Alpha = 0.5

Orange: Train | 76.91

Red: Valid | 73.42

Don’t work with videos.

Conclusion

Conclusion (Real) and Future Directions

• We verified that the model is indeed looking at the right features through our visualization and exploration.
• The model is prone to adversarial attacks – both targeted and untargeted. We were able to get the model to say what we wanted to.
• Naive adversarial training provides slight improvement, but this area warrants further exploration.

Experiment: WER vs Epsilon Results

Experiment: RickRolling AVHubert

Experiment: Adversarial Examples

Prediction

which is that it sounds kind of cool and then like a form like a melody around in and that's a lot of fun like i would

Prediction

i said hey aren't it really occurred it's like yelling out of football and martin luther king and yelling al qaeda and i'd like to site on her philosophy and you

Prediction

Never gonna give you up, Never gonna let you down

Experiment: Train / Val Curves

Training and Validation Accuracy for Standard Training

Pink: Train | 72.92

Green: Valid | 69.22

Training and Validation Accuracy for Adversarial Training with Alpha = 0.5

Orange: Train | 76.91

Red: Valid | 73.42