Contrastive Language-Audio Pre-training:

Expanding CLIP to Audio

Yusong Wu*, Tianyu Zhang*, Ke Chen*, Richard Vencu, Christoph Schuhmann

*: equal contribution

Collaborators

CLIP: Unsupervised Multi-modal Contrastive learning

CLIP: SOTA Performance

• Dominate ResNet-101 on ImageNet
• Robust on various domains
• More natural object detection

CLIP: Fundation of Other Models

Expanding CILP to Other Modalities

• Human perceive more than 2 modalites
• Learning multiple modalites enhance each other
• Enable audio-related down-stream tasks

Expanding CILP to Other Modalities

• Expand CLIP to audio

Overview

• Larger dataset with natural text

Overview

• Larger dataset with natural text
• CLAP model with audio encoder

Overview

• Larger dataset with natural text
• CLAP model with audio encoder
• Experiments and takeaways

Previous Works have limited Data

• Train on small dataset
• Or train with only label data

Cat, Dog, Cars, …

Ours: Mixture Dataset with Natural Text

• ~2,000,000 samples, 5893 hours
• audio-text pair & audio-label pair

Cat, Dog, Cars, …

A muddled noise of broken channel of the TV. …

Convert Lable to Text Prompt

For label data, construct prompt:

Label: cat, dog, car

Text prompt: Sound of cat, dog, and car.

CLAP: Adding Audio Encoder to CLIP

MLP

MLP

Text encoder

Audio encoder

Cross-entropy Loss

Cross-entropy Loss

Frozen Parameters

Training Parameters

Losses

Text / Audio Networks

Audio Encoder

• CNN-based audio encoder:

Pretrained Audio Neural Networks (PANN)

81M parameters

Audio Encoder

• CNN-based: Pretrained Audio Neural Networks (PANN), 81M parameters
• Transformer-based audio encoder:

Hierarchical Token-Semantic Audio Transformer (HTS-AT), 28M parameters

Data Loading

• Audio: 10s segement
• Random select segement for longer audio
• Pad 0 for shorter audio

Data Loading

• Audio: 10s segement
• Random select segement for longer audio
• Pad 0 for shorter audio
• Input representation: Mel-spectrogram, 64 freq bins
• Sample rate: 48kHz, Frame size: 480, → 1000 input frames

Data Loading

• Audio: 10s segement
• Random select segement for longer audio
• Pad 0 for shorter audio
• Input representation: Mel-spectrogram, 64 freq bins
• Sample rate: 48kHz, Frame size: 480, → 1000 input frames
• Text:
• For more than one captions, random select one
• Tokenized by sub-word tokenizer (same as GPT)

Multi-GPU Training with Efficient Dataloader

• Server: AWS instance with 8 A100 GPU

Multi-GPU Training with Efficient Dataloader

• Server: AWS instance with 8 A100 GPU
• Pre-process data into unified format

Multi-GPU Training with Efficient Dataloader

• Server: AWS instance with 8 A100 GPU
• Pre-process data into unified format
• Pack samples into tars read by webdataset

Multi-GPU Training with Efficient Dataloader

• Server: AWS instance with 8 A100 GPU
• Pre-process data into unified format
• Pack samples into tars read by webdataset
• Avoid process data in dataloader or training loop

Training

• Keep most hyperparameters same as CLIP
• 1e-3 learning rate, warm-up, cosine learning rate decay
• AdamW optimizer, 0.1 weight decay

Evaluation Metrics

• Text-to-Audio R@1 / R@5 / R@10

Evaluation Metrics

• Text-to-Audio R@1 / R@5 / R@10: larger is better

Top 1/5/10 Accuarcy

Evaluation Metrics

• Text-to-Audio R@1 / R@5 / R@10: larger is better
• Audio-to-Text R@1 / R@5 / R@10: larger is better

Evaluation Metrics

• Text-to-Audio R@1 / R@5 / R@10: larger is better
• Audio-to-Text R@1 / R@5 / R@10: larger is better
• Validation Loss

Exp set 1: Small dataset overfit?

• Train loss vs. Val loss: clearly overfit

Exp set 1: Small dataset overfit?

• Train loss vs. Val loss: clearly overfit

Exp set 1: Small dataset overfit?

• Retrieval-based eval metric shows trend of quasi-monotonic increasing

Exp set 2: Large dataset vs small dataset

• Introduced a 30x larger dataset: audioset
• Less overfitting seen in val loss on large dataset

Exp set 2: Large dataset vs small dataset

• Introduced a 30x larger dataset: audioset
• Metric increases fast at first small dataset, but plateaus.
• Metric continue to increase in large dataset.

Exp set 3: Transformer > CNN

• Although differnt in size, transformer outperforms CNN at last

Exp set 3: Transformer > CNN

• Although differnt in size, transformer outperforms CNN at last
• Transformer: faster run time

Exp set 4: Model Scaling wrt. Dataset Size

• 1x, 0.5x, 0.25x largest dataset
• Smaller dataset overfits more in val loss

Exp set 4: Model Scaling wrt. Dataset Size

• 1x, 0.5x, 0.25x largest dataset
• Larger dataset has better performance & growth rate

Conclusion

• Collected a large audio dataset
• Successfully build and trained a CLAP model
• Premitive experiment results

Limitations and Future works

• Collect more dataset
• More evaluation and down-stream task
• More experiments
• Better audio encoder