PAT: Parameter-Free Audio-Text Aligner to Boost Zero-Shot Audio

Classification

Ashish Seth, Ramaneswaran Selvakumar, Sonal Kumar, Sreyan Ghosh,

Dinesh Manocha

(University Of Maryland, College Park)

​

Motivation

• Limitations of Simplistic Prompts: Current Audio-Language Encoders (ALEs) rely on basic prompts like "The sound of <label>" during zero-shot evaluation, which limits their ability to capture the complexity of real-world sounds, leading to suboptimal performance.
• Loss of Discriminative Power: Zero-shot transfer commonly aligns audio and text representations via cosine similarity, using average pooling for audio representations. This process causes information loss, weakening the model's discriminative power for performing audio classification.

Problem Statement

“How can we adapt Audio-Language Models to out-of-distribution audio classification tasks in parameter/training free fashion”

Key contributions

• We propose PAT, a novel approach to improve zero-shot audio classification performance in a training-free fashion. PAT introduces a cross-modal interaction approach aimed at improving audio-text alignment by enhancing both audio and textual representations in a zero-shot setting.�
• We evaluate PAT across multiple ALEs on 18 audio classification datasets and show that PAT achieves 0.42%–27.0% improvement over our baselines.�
• We further investigate PAT’s robustness to noisy audio to show that PAT consistently outperforms our baselines under varied noise augmentation settings.

​

Methodology (Date Store)

• To perform prompt ensemble, we first create a novel datastore comprising of 400 semantically and linguistically diverse prompts.
• We use GPT-4o to generate 600 diverse prompts by providing diverse templates and generation guidelines.
• Finally we conduct manual filtering to select 400 prompts by discarding those that are inaccurate or repetitive.

Guidelines for prompt generation

Methodology (Weighted Prompt Ensemble)

• Weighted Prompt Ensemble: Computes a score for each prompt based on ALEs' feedback, creating a weighted ensemble of text embeddings with class labels—achieving adaptive prompt weighting without labeled data or extra training.

​

​

Methodology (Weighted Prompt Ensemble)

• For each prompt, we first compute a cumulative sum of max prediction logits as defined below:

• Next, we normalize these weights using softmax, and apply them further to perform weighted prompt ensemble.

Methodology (Cross Modal Alignment)

• Cross-Modal Aligner: Uses parameter-free attention to align audio and enriched text representations, enabling enhanced frame-level audio understanding and zero-shot classification through cosine similarity

​

​

Results (Main Results)

Performance comparison between PAT and vanilla zero-shot classification (ZS) across six ALEs and 16 diverse audio classification tasks, including 10 sound and 8 music datasets. The best scores for each ALE are bolded. Overall, PAT outperforms vanilla ZS, achieving improvements ranging from 0.42% to 27%.

Results (Ablations)

Zero-shot performance evaluation of MSCLAP-23 using PAT across 16 audio classification tasks under noisy conditions. Each audio sample undergoes five different types of audio augmentations. PAT outperforms vanilla zero-shot (ZS) classification, achieving an absolute improvement of 0.10%–11.15%.

Thank You:)