1 of 69

Advancing Audio Processing in the Age of Large Language Models

Sreyan Ghosh

Ph.D. Student

GAMMA Lab and PIRL Lab

University of Maryland, College Park

1

​

Examining Committee: Dr Dinesh Manocha

Dr Ramani Duraiswami

Dr Nirupam Roy

Dr Mohit Iyyer

2 of 69

2

Audio as a Modality

�

Humans rely heavily on auditory information in daily life. Auditory cues enhance situational awareness and communication.

Auditory cues include both verbal and non-verbal audio.

Verbal audio includes spoken speech and helps humans communicate. 

Non-verbal audio includes the non-verbal speech (e.g., sneeze, cough, etc.) and non-speech sounds (e.g., environmental sounds and music). Non-verbal cues are as important as verbal cues

3 of 69

3

Audio as a Modality in AI

�

  • AI agents & audio perception - Audio is essential for creating intelligent, context-aware AI agents. A lot of tasks require agents to understand and reason about the audio cues in the environments.

​

  • Existing tasks can greatly benefit from audio cues -movie summarization, object detection, etc.

​

  • Audio can improve immersiveness of generated content - enhances immersiveness, creating richer and more engaging experiences that complement visual and textual information.

��

Generated videos can be more immersive with videos!

An example of improving imitation learning in robots with auditory cues. Example from ManiWAV (Liu et al., 2024)

Agents that can hear and speak understand and communicate better with humans.

Movie summarization using only videos misses important information hidden in sound effects and human communication!

4 of 69

4

Examples of Improving AI systems with Audio Perception

�

​

​

​

​

  • Visual-only tasks have shown to benefit with audio integration
    • Audio-Visual Object detection
    • Video Question-Answering with audio cues
    • Robot Navigation with audio-visual cues

​

​

​

​

​

​

��

Audio-Visual Scene Captioning

Caption w/o Audio: A black dog lies on the sofa.

Caption with Audio: In the living room, a black dog lies on the sofa barking, with the sound of a police car in the background.

GPT4o can now hear and speak! Audio integration has brought a leap in GPT's capabilities. 

Egocentric Audio-Visual Object Detection.

(Huang et al., 2023)

Robot navigation with audio-visual cues.

(Huang et al., 2023)

​

  • Scene understanding has improved incredibly with audio integration:
    • Improved ego-centric perception
    • Improved scene captioning
  • GPT-4o is integrated with hearing and speaking capabilities.

5 of 69

5

​

  • Audio Understanding Systems: 
    • Encoders such as Audio Spectrogram Transformer (Gong et. Al 2021), HTSAT (Chen et al. 2022), etc.
    • Audio-Language models such as CLAP (Elizalde et. al 2022), CompA (Ghosh et al. 2023), etc.
    • Audio LLMs such as LTU (Gong et. al 2023), GAMA (Ghosh et al. 2024), etc.

​

​

​

  • Audio Generation Systems:
    • Diffusion-based Acoustic Event Generation Models such as AudioGen (Kreuk et al. 2022), etc.
    • Speech Generation Models such as VALLE (Wang et al. 2023), etc.
    • Audio Story Telling Systems such as WavJourney (Liu et al. 2023), etc.�

Recent Advances in Audio Processing

​

�

LLMs that can reason about input speech and sounds and answer queries (GAMA, Ghosh et al. 2024 ).

An amalgamation of understanding and generation for audio storytelling (WavJourney, Liu et al., 2023)

Diffusion models that can generate audio from a query text (AudioGen, Kreuk et al., 2022).

Recreate a gentle rainfall with distant thunder.

6 of 69

​

  • Limited Audio Datasets: Audio data is scarce compared to other modalities, with minimal availability for certain tasks. Many datasets are restricted or lack real-world diversity.
    • Lack of Reasoning-Centric Data: Most datasets focus on recognition rather than complex reasoning. Unlike language and vision, distilling larger models for reasoning remains a challenge.
    • Lack of Long Audio Processing Data: Open-source datasets typically contain 5–10s clips, with no datasets dedicated to long-form sounds and music beyond ASR.

​

  • Weak Audio Representations:
    • State-of-the-art audio encoders achieve ~50% on AudioSet, far below 95%+ on ImageNet.
    • Models struggle with compositional audio structure (e.g., event order) and linguistic variations (e.g., "chopper" vs. "helicopter").
    • Speech, sounds, and music are still treated separately rather than holistically.

​

  • Lack of Good Evaluation Benchmarks: Most benchmarks remain recognition-focused and are clean and simple, limiting progress in real-world understanding and reasoning tasks. 25+ reasoning benchmarks exist for vision and language, but only a few for audio.

​

​

​

​

​

​

​

​

​

​

6

Current Challenges in Audio Processing

�

Complex Instruction: From the speech and impact sounds, deduce the size and characteristics of the room in which these events occur.

Simple Event Detection: What are the acoustic events in the given audio.

Non-verbal Audio Datasets

Largest Dataset -  ~1k Hours

​

Vision Datasets

Largest Dataset -  5B+ I-T pairs

Language Datasets

Largest Dataset - Billions of tokens

Speech Datasets

Largest Dataset -  60k+ Hours

7 of 69

7

  • Compositional Understanding for Audio Processing: Real-world audio is often compositionally complex. Understanding and generative models trained on clean audios find it difficult to generalize to real-world audios.
    • More compositional audio data
    • Better model architectures
    • Better training algorithms
    • Better audio representations for generative models

​

​

  • Advanced Reasoning in Audio: Current state of the art models struggle to answer complex questions on audio which require advanced reasoning and answering using world knowledge.
    • Advanced evaluation benchmarks
    • Better reasoning-centric data

​

​

  • Long and Multi Audio Processing: Most audio models can perceive a single audio with a max of 30 secs in length (with 10 secs being the average).
    • Long audio encoders
    • Long audio representation learning
    • Long audio datasets and benchmarks
    • Multi-audio reasoning strategies

Caption: A dog barking followed by a person speaking – generated by Stable Audio

How are the state-of-the-art audio models doing?

Question: What indicates a sudden moment of fear in the audio?

​

Answer by LTU: There is no such moment in the audio.

​

Question: Based on various sounds, infer what the man might be announcing.

​

Answer by LTU (Gong et. al): It is not possible to infer.

8 of 69

(Large) Audio Language Models

​

​

  • Post-LLM Era: LALMs
    • LALMs integrate audio encoders with LLMs, unlocking deeper audio perception and reasoning.
    • Examples: GAMA, LTU, Qwen, AudioFlamingo, SALMONN.
    • Capable of captioning, reasoning and open-ended QA.

8

CLAP model learned with contrastive audio-text pairs.

LALMs with audio encoders integrated

with LLMs.

  • What are Audio-Language Models (ALMs)?
    • Audio processing is challenging due to data scarcity, but language is abundant.
    • ALMs learn a shared latent space between audio-language modalities.
  • Pre-LLM Era: CLAP & Contrastive Learning
    • CLAP (inspired by CLIP) was a key pre-LLM ALM, using contrastive learning to align audio and text.
    • Useful for classification, audio-to-text retrieval, and text-to-audio retrieval.

9 of 69

9

Current State-of-the-Art Audio (Large) Language Models

Open Source

Proprietary

  • Gemini 1.5
  • GPT-4o

Open Access

  • CLAP
  • LAION-CLAP
  • ReCLAP
  • SALMONN

​

​

10 of 69

CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models (ICLR 2024)

10

  • Current CLAP-like Audio-Language models lack compositional reasoning.

​

Current ALMs are not good at compositional reasoning.

Model performance does not drop on current retrieval benchmarks with words shuffled.

  • Compositional Reasoning: Understanding the relationship between text in captions and the corresponding content of the audio is a fundamental goal of audio processing. Different word orders should correspond to differently perceived audio.
  • Current benchmarks are insufficient for evaluating compositional reasoning.

11 of 69

CompA: CompA Benchmark

We propose CompA, the first suite of benchmarks for evaluating compositional reasoning in ALMs. CompA-order evaluates an ALM's capability to understand the order of occurrence between multiple acoustic events in an audio. CompA-attribute evaluates an ALM’s capability to understand attribute-binding for multiple acoustic events in an audio.

11

A child sneezes, and an adult laughs.

A child laughs, and an adult sneezes.

A tiger growls followed by people talking.

People talking followed by tiger growling..

12 of 69

CompA: CompA Benchmark Evaluation

12

Text score.

Audio score.

Combined score.

Current models perform poorly.

13 of 69

CompA: CompA-661k

13

Overview of CompA-661k, combining open-source datasets with our proposed CompA-AudioSet.

CompA-AudioSet improves compositional balance compared to LAION Audio-630k.

14 of 69

CompA: CompA-CLAP

14

Compositionally-Aware Hard Negative Training: We introduce two new algorithms to enhance compositional representation learning in CLAP. First, we modify the CLAP objective to incorporate compositionally-aware hard negatives. In this approach, each audio sample in the batch is assigned additional hard negatives that are not shared across other batches, improving the model’s ability to distinguish fine-grained compositional variations.

15 of 69

15

Modular Contrastive Learning: We introduce a modular template-based approach that generates compositional audio-caption pairs from single acoustic events, aligning each audio with positives and compositionally-aware hard negatives. Using AudioSet snippets and LLM-generated scenes, we create high-quality synthetic data. Our training strategy optimizes contrastive loss to capture fine-grained compositional relationships

CompA: CompA-CLAP

Synthetic data generation strategy.

16 of 69

CompA: Results on Benchmark Datasets

16

Result comparison on AudioCaps/Clotho retrieval benchmarks.

Result comparison on zero-shot audio classification benchmarks.

17 of 69

CompA: Results on CompA-order and CompA-attribute

17

Result comparison on our proposed CompA benchmarks.

18 of 69

  • Large Audio-Language Models (LALMs) struggle to respond to user queries that require complex reasoning.
  • Most LALMs just use a single audio encoder with a linear layer to integrate audio representations to LLMs. This leads to limited audio understanding and increases hallucinations. 
  • GAMA is developed to improve both aspects!

18

GAMA: A General-purpose Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities (EMNLP 2024 Oral)

​

​

Comparison of GAMA with LTU (Gong et al., 2023) on a query about an audio that involves complex reasoning.

19 of 69

19

To improve audio perception, GAMA integrates multiple audio representations with a large language model to allow it to perceive diverse knowledge about an input audio.

​

​

​

​

​

​

​

​

​

​

​

​

​

Illustration of the GAMA architecture.

GAMA: Model Architecture

20 of 69

20

To improve audio perception, GAMA integrates multiple audio representations with a large language model to allow it to perceive diverse knowledge about an input audio.

    • We employ AST as our audio encoder that encodes mel-spectrograms.

​

​

​

​

​

​

​

​

​

​

​

​

Illustration of the GAMA architecture.

GAMA: Model Architecture

21 of 69

21

To improve audio perception, GAMA integrates multiple audio representations with a large language model to allow it to perceive diverse knowledge about an input audio.

    • We employ AST as our audio encoder that encodes mel-spectrograms.
    • We introduce multi-layer aggregator to improve fine-grained understanding of the audio.

​

​

​

​

​

​

​

​

​

​

​

​

​

Illustration of the GAMA architecture.

GAMA: Model Architecture

22 of 69

22

To improve audio perception, GAMA integrates multiple audio representations with a large language model to allow it to perceive diverse knowledge about an input audio.

    • We employ AST as our audio encoder that encodes mel-spectrograms.
    • We introduce multi-layer aggregator to improve fine-grained understanding of the audio.
    • We train a custom Audio QFormer to improve semantic understanding of the input audio.

​

​

​

​

​

​

​

​

​

​

​

​

​

Illustration of the GAMA architecture.

GAMA: Model Architecture

23 of 69

23

To improve audio perception, GAMA integrates multiple audio representations with a large language model to allow it to perceive diverse knowledge about an input audio.

    • We employ AST as our audio encoder that encodes mel-spectrograms.
    • We introduce multi-layer aggregator to improve fine-grained understanding of the audio.
    • We train a custom Audio QFormer to improve semantic understanding of the input audio.
    • We introduce soft prompt to provide high-level semantic evidence of acoustic events.

​

​

​

​

​

​

​

​

​

​

​

​

​

Illustration of the GAMA architecture.

GAMA: Model Architecture

24 of 69

  • GAMA is trained in 2 stages. First, it is pre-trained on a large-scale dataset with audio-language pairs. Next, it is instruction-tuned on CompA-R, a novel instruction tuning dataset.

24

GAMA: CompA-R Reasoning Dataset

Our proposed pipeline to synthesize CompA-R.

​

  • What is CompA-R?: CompA-R (Instruction-Tuning for Complex Audio Reasoning) is a dataset designed to train models to understand and reason about complex audio scenarios. We also propose CompA-R-test, a novel benchmark to evaluate LALMs on complex reasoning abilities.

25 of 69

25

GAMA: Results on Benchmark Datasets

Comparison of GAMA with baselines on evaluation datasets described on close-ended general audio and music understanding benchmarks.

26 of 69

26

Comparison of GAMA with other baselines on open-ended AQA on OpenAQA, complex open-ended AQA on CompA-R- test and Dense Captioning on 500 instances from AudioCaps and Clotho.

GAMA: Results on Reasoning Datasets

27 of 69

27

GAMA: Results on Reasoning Datasets

Comparison of GAMA with other LALMs on deductive reasoning benchmark (Deshmukh et al., 2024).

Comparison of GAMA with other LALMs on object hallucination benchmark (Kuan et al., 2024).

28 of 69

28

GAMA: Qualitative Examples

29 of 69

29

We present MMAU, the most comprehensive audio understanding and reasoning benchmark. MMAU comprises 10k carefully curated audio clips paired with natural language questions and answers spanning speech, environmental sounds, and music.

MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (ICLR 2025 Spotlight)

​

​

  • Morris et al., 2024 define Artificial General Intelligence (AGI) as a system that performs at the “90th percentile of skilled adults” across a wide array of tasks.
  • Current audio benchmarks fall short of this standard. Tasks such as speech recognition do not demand the expertise of skilled humans and can often be performed by young children (Lippmann, 1997; Gerhardstein & Rovee-Collier, 2002)
  • MMLU (for language) and MMMU (for vision) have pushed the boundaries of model capabilities, prompting incremental improvements. Nothing as such exists in audio.

30 of 69

30

MMAU: Skill Distribution

​

MMAU is composed of questions that challenge models no 27 distinct skills, 16 Reasoning (right) and 11 Information Extraction Tasks (left).

�

31 of 69

31

MMAU: Benchmark Examples

​

Skill: Temporal Event Reasoning ��Question: Identify the total number of drumbeats in the audio. �Choices: A. 2 B. 4 C. 5 D. 3

Skill: Acoustic Sound Reasoning. ��Question: In the given audio, what might cause the dog’s reaction? �Choices: (A). A loud noise followed by a siren. (B). A low clicking noise followed by repeated clicking. ….

Skill: Event-Based Knowledge Retrieval ��Question: Who was the scientist behind the discovery mentioned by the speaker?�Choices: A. Madam Curie B. Albert Einstein C. CV Raman D. Issac Newton

Skill: Emotion Flip Detection��Question: From the given conversation, Identify the speakers that showed emotion flip.�Choices: A. both speakers B. first speaker C. second speaker D. none of the speaker

Skill: Socio-cultural Interpretation��Question: In which cultural setting would the music in the audio most likely be performed?�Choices: A. Western classical concert hall B. Indian classical music festival C. Modern pop concert D. Jazz club

Skill: Musical Texture Interpretation��Question: What is the main characteristic of the male voice in the audio?�Choices: A. Soft and mellow B. Loud and soulful C. High-pitched and fast D. Monotone and slow

Sound

Speech

Music

32 of 69

32

MMAU: Scores for Frontier Models

​

Most models perform poorly on MMAU, thereby highlighting significant gap.�

33 of 69

33

MMAU: Skill-Specific Performance

​

Accuracy distribution for Gemini 2.0 Flash across easy, medium, and hard questions, categorized by skill type. The graph highlights how LALMs excel in some skills across all difficulty levels (e.g., Phonemic Stress Pattern Analysis) but struggle with others (e.g., Temporal Reasoning) regardless of difficulty.

34 of 69

34

MMAU: Where are they falling short?

​

Distribution of human-annotated error types across 500 instances for Qwen2-Audio-Instruct (Left) and Gemini 2.0 Flash (Right). The dominant error type is Perceptual Errors.

35 of 69

35

Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data (ICLR 2025)

​

​

  • Synthetic data enables the creation of tailored training datasets, mitigating the impact of noise in web datasets and serving as a valuable resource in low-resource scenarios.

​

  • The use of synthetic data to improve understanding and reasoning is booming in language and vision (e.g., Phi4, GPT-4o).

​

  • There is lack of foundational audio generation models.

​

  • Existing data augmentation methods (noise augmentation or masking) do not capture the true diversity present in real-world audios.

​

  • We propose Synthio, a novel synthetic data generation technique to generate diverse and consistent synthetic audios to improve audio classification.

​

Traditional augmentation, such as SpecAug, degrades performance on small-scale datasets. Naive augmentation outperforms traditional methods significantly but plateaus with higher sample counts. Synthio further enhances performance by generating consistent and diverse synthetic data.

36 of 69

36

Synthio: Motivation

Problem 1 – Alignment: When prompted with “sound of a bus” for the category bus in the TUT-Urban dataset, the generated audio may not reflect the typical bus sounds in European cities (where TUT was recorded), as bus sounds can vary by region, with some featuring loud engines and dense crowds while others have quieter engines and sparse crowds.

Can we just fine-tune the text-to-audio model? -- Fine-tuning on small datasets lead to overfitting and generation of audios that sound similar.

Sound of a bus from the TUT Urban dataset.

Sound of a bus from a text-to-audio model trained on AudioSet.

Problem 2 – Lack of feature diversity: When prompted with “Sound of a park’”, we observed that 9 out of 10 times, the model generated the sound of children playing as part of the generated audio.

Two sounds with different random seeds, CFG and inference steps.

37 of 69

37

Synthio: Aligning the Text-to-Audio Model using Preference Optimization

  • Step 1: Constructing the Preference Dataset

​

    • Generate template-based captions for each instance (e.g., "Sound of a [label]").
    • Prompt the T2A model multiple times per instance, pairing generated audio with ground-truth audio, treating generated as "loser" and ground-truth as "winner.”

​

  • Step 2: Preference Optimization with DPO (Rafailov et. al)

​

    • Train the T2A model on the preference dataset using DPO.
    • The optimized model improves alignment with ground-truth audio.

We align the T2A model with the small-scale dataset using DPO, ensuring generated audio matches its acoustic characteristics.

38 of 69

38

Synthio: Generating Diverse Synthetic Augmentations

We align the T2A model with the small-scale dataset using DPO, ensuring generated audio matches its acoustic characteristics.

39 of 69

39

Synthio: Generating Diverse Synthetic Augmentations

Step 1: Generate captions for all audios using an audio captioning model.

40 of 69

40

Synthio: Generating Diverse Synthetic Augmentations

Step 2: Extract existing acoustic components (events, attributes)

“Children playing and shouting loudly in a metro station” – {children playing, shout, loud}

41 of 69

41

Synthio: Generating Diverse Synthetic Augmentations

Step 3: Generate new captions for labels

“People speaking in a metro station”

42 of 69

42

Synthio: Generating Diverse Synthetic Augmentations

43 of 69

43

Synthio: Results

Results on benchmark datasets with just 50 and 100 gold training samples. Synthio shows huge performance gains compared to our baselines.

44 of 69

44

Synthio: How diverse and consistent are the augmentations?

Spectral and pitch feature comparison between Synthio-generated audios and real audios. Synthio effectively preserves key audio characteristics, ensuring consistency with real data.

45 of 69

45

Synthio: Extra Analysis

Results for audio captioning, a more complex task than classification. Synthio consistently improves performance.

Results for scaling synthetic data with Synthio. Synthio maintains better scalability than our baselines.

46 of 69

46

Failing Forward: Improving Generative Error Correction for ASR with Synthetic Data and Retrieval Augmentation (ACL 2025 Submission)

Example results (truncated). DARAG improves performance by substantial margins.

47 of 69

47

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (ICML 2025 Submission)

​

​

Audio Flamingo 2 (AF2) versus previous SOTA on a number of audio understanding and reasoning benchmarks. AF2 outperforms all models while being the smallest.

Audio Flamingo 2 is a SOTA LALM that focuses on 2 main aspects:

​

  • Improving audio understanding and reasoning
    • We develop AF-CLAP, a custom audio encoder for robust representations
    • Generate synthetic reasoning data at scale

​

  • Introduce Long Audio Understanding (30 seconds – 5 minutes)
    • We introduce LongAudio, the first long audio reasoning dataset
    • We propose several novel training strategies

​

48 of 69

48

AF2: Overall Architecture

Audio Flamingo 2 is based on the cross-attention architecture, where we condition our custom CLAP. The model is trained on 3 stages of training and can perceive audios up to 5 minutes in length.

49 of 69

49

AF2: Improved Audio Representations with AF-CLAP

Overall AF-CLAP training pipeline.

We focus on two primary aspects in AF-CLAP :

  1. We scale training data to 8M audio-caption pairs:
    1. Generate new data from long videos, inspired from Venkataramanan et al., 2024
  2. We propose a new training objective
    • Improve linguistic invariance
    • Improve compositional reasoning

​

​

​

​

​

    • Improve the CLAP training objective

​

​

​

50 of 69

50

AF2: AudioSkills – Large-Scale Synthetic Reasoning Data

We propose AudioSkills, the 1st and the largest audio reasoning data with ~4.2M AQA pairs. AudioSkills comprises of skill-wise reasoning data with seven distinct skills.

51 of 69

51

AF2: LongAudio – A Long Audio Understanding Dataset

The proportion of video categories and distribution of durations for the LongAudio dataset with 262,928 unique AQA and 80k unique audios. We target captioning and reasoning tasks.

52 of 69

52

AF2: LongAudio – A Long Audio Understanding Dataset

The pipeline for generating LongAudio. The process begins by segmenting the long video into short video and audio clips, each ~10 seconds. These clips are individually annotated with captions. Subsequently, an LLM is employed to generate question-and-answer pairs based on the captions of these clips. A subset of the data goes through expert review to construct LongAudioBench.

53 of 69

53

AF2-3B: Results on Benchmark Datasets

Results on Audio Classification and Captioning Tasks. AF2 outperforms larger models on most of these tasks.

54 of 69

54

AF2-3B : Results on Reasoning Datasets

Results on expert reasoning tasks. AF2 outperforms all models by significant margins.

55 of 69

55

AF2: Result Analysis

Performance comparison of AF2 on different LLM sizes, w/ and w/o AudioSkills. Improving the quality of training data proves to be better than scaling LLM size.

Our proposed 3-stage training strategy proves to the most effective technique for effective reasoning.

56 of 69

56

AF2: Qualitative Results

57 of 69

57

AF2: Qualitative Results

58 of 69

  • Combining audio generation and understanding
    • Research Question 1: Can audio generation and understanding help each other?
    • Research Question 2: How to effectively build models that can both understand and input audio and generate one?

​

  • Thinking Models: 
    • Research Question: Scaling inference-time compute for audio generation and understanding by moving to the thinking paradigm

​

  • Making Audio-Language Models more efficient:
    • Research Question: How far can we go with small language models?

​

  • Audio-visual complex reasoning: 
    • Research Question 1: How do we generate instruction-tuning datasets for complex reasoning over audio-visual cues?
    • Research Question 2: Leveraging findings from our audio work, how do we build effective architectures for effective audio-visual reasoning?

​

​

58

Future Work

59 of 69

59

Pre-UMD

Fall 2022

Spring 2023

Summer 2023

Fall 2023

Spring 2024

Summer 2024

Fall 2024

Low Resource ASR

Audio SSL

Audio SSL

Synthetic Data

Synthetic Data

Synthetic Data

Synthetic Data

Audio-Language Models

Synthetic Data

Audio-Language Models

Instruction Tuning

Synthetic Data for Audio

Audio-Language Models

Synthetic Data for Audio

Timeline

Spring 2025

Fall 2025

Spring2026

Large Audio-Language Models

Long Speech

Large Omni Models

​

Defense

​

ICASSP ’23

SIGIR ‘23

ACL ’23, ICCV ’23, EMNLP ’23

InterSpeech ‘23

Adobe

ICLR ‘24, CVPR ’24, NAACL ‘24

ICML ’24, EMNLP ‘24

Microsoft

NVIDIA�ICLR ‘25

ICML 25 (sub), ACL ’25 (sub)

Summer 2025

Defense

​

NVIDIA

60 of 69

Thanks to all our sponsors and collaborators!

60

NVIDIA 2025 Graduate Fellowship

Graduate School RA Award 2023

61 of 69

61

Papers (Audio Processing)

  1. GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities�Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, Dinesh Manocha�EMNLP 2024
  2. Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data�Sreyan Ghosh, Sonal Kumar, Zhifeng Kong, Rafael Valle, Bryan Catanzaro, Dinesh Manocha�ICLR 2025
  3. ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds�Sreyan Ghosh, Sonal Kumar, Chandra Kiran Reddy Evuru, Oriol Nieto, Ramani Duraiswami, Dinesh Manocha�ICASSP 2025
  4. LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition�Sreyan Ghosh, Sonal Kumar, Ashish Seth, Purva Chiniya, Utkarsh Tyagi, Ramani Duraiswami, Dinesh Manocha�InterSpeech 2024
  5. AV-RIR: Audio-Visual Room Impulse Response Estimation�Anton Ratnarajah, Sreyan Ghosh, Sonal Kumar, Purva Chiniya, Dinesh Manocha�CVPR 2024
  6. CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models�Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, C. K. Evuru, Oriol Nieto, Dinesh Manocha�ICLR 2024
  7. MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark�S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, Dinesh Manocha�ICLR 2025
  8. RECAP: Retrieval-Augmented Audio Captioning�Sreyan Ghosh, Sonal Kumar, Chandra Kiran Reddy Evuru, Ramani Duraiswami, Dinesh Manocha�ICASSP 2024
  9. EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning�Ashish Seth, Ramaneswaran S, S Sakshi, Sonal Kumar, Sreyan Ghosh, Dinesh Manocha�EMNLP 2024
  10. Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognition�Ashish Seth, Sreyan Ghosh, S. Umesh, Dinesh Manocha�ICASSP 2024

​

62 of 69

62

Papers (Audio Processing)

  1. FusDom: Combining In-Domain and Out-of-Domain Knowledge for Continuous Self-Supervised Learning�Ashish Seth, Sreyan Ghosh, S. Umesh, Dinesh Manocha�ICASSP 2024
  2. AdVerb: Visually Guided Audio Dereverberation�Sanjoy Chowdhury, Sreyan Ghosh, Subhrajyoti Dasgupta, Anton Ratnarajah, Utkarsh Tyagi, Dinesh Manocha�ICCV 2023
  3. MMER: Multimodal Multi-task Learning for Speech Emotion Recognition�Sreyan Ghosh, Utkarsh Tyagi, S Ramaneswaran, Harshvardhan Srivastava, Dinesh Manocha�InterSpeech 2023
  4. MAST: Multiscale Audio Spectrogram Transformers�Sreyan Ghosh, Ashish Seth, S. Umesh, Dinesh Manocha�ICASSP 2023
  5. SLICER: Learning universal audio representations using low-resource self-supervised pre-training�Ashish Seth, Sreyan Ghosh, S. Umesh, Dinesh Manocha�ICASSP 2023
  6. End-to-end Named Entity Recognition from English Speech�Hemant Yadav, Sreyan Ghosh, Yi Yu, Rajiv Ratn Shah�InterSpeech 2020
  7. ProSE: Diffusion Priors for Speech Enhancement�Sonal Kumar, Sreyan Ghosh, Utkarsh Tyagi, Purva Chiniya, Anton Jeran Ratnarajah, Chandra Kiran Reddy Evuru, Ramani Duraiswami, Dinesh Manocha�NAACL 2025

​

63 of 69

63

Papers (NLP and Vision)

  1. Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs�Sreyan Ghosh, C. K. Evuru, Sonal Kumar, Utkarsh Tyagi, O. Nieto, Z. Jin, Dinesh Manocha�ICLR 2025
  2. A Closer Look at the Limitations of Instruction Tuning�Sreyan Ghosh, C. K. Evuru, Sonal Kumar, Ramaneswaran S, D. Aneja, Z. Jin, R. Duraiswami, Dinesh Manocha�ICML 2024
  3. ABEX: Data Augmentation for Low-Resource NLU via Expanding Abstract Descriptions�Sreyan Ghosh, Utkarsh Tyagi, Sonal Kumar, Chandra Kiran Reddy Evuru, Ramaneswaran S, S Sakshi, Dinesh Manocha�ACL 2024
  4. ASPIRE: Language-Guided Augmentation for Robust Image Classification�Sreyan Ghosh, C. K. Evuru, Sonal Kumar, S. Sakshi, Utkarsh Tyagi, Dinesh Manocha�ACL 2024 (Findings)
  5. DALE: Generative Data Augmentation for Legal NLP�Sreyan Ghosh, C. K. Evuru, Sonal Kumar, S. Sakshi, Utkarsh Tyagi, Dinesh Manocha�EMNLP 2023
  6. CoSyn: Detecting Implicit Hate Speech in Online Conversations Using a Context Synergized Hyperbolic Network�Sreyan Ghosh, Manan Suri, Purva Chiniya, Utkarsh Tyagi, Sonal Kumar, Dinesh Manocha�EMNLP 2023
  7. ACLM: A Selective-Denoising based Generative Data Augmentation Approach for Low-Resource Complex NER�Sreyan Ghosh, Utkarsh Tyagi, Manan Suri, Sonal Kumar, S Ramaneswaran, Dinesh Manocha�ACL 2023

64 of 69

Thank You!��Questions?

64

65 of 69

65

GAMA: Complex QA Examples

GAMA (Ghost et al., 2024)

The woman is likely bathing her child, as indicated by the sound of a bathtub filling up with water.

LTU (Gong et al., 2023)

The most probable activity occurring during the audio clip is a family or group of people having fun together, possibly playing with babies or toddlers. The mechanical sounds could be from a playground or other outdoor equipment.

Pengi (Deshmukh et al., 2024)

It is a combo.

SALMONN (Tang et al., 2023)

Based on the variety of sounds including mechanisms, generic impact sounds, and water splashes, as well as the presence of baby laughter, the most probable activity occurring during the audio clip is likely a bath or a water-related activity.

GAMA (Ghost et al., 2024)

The tap dancing is likely performed by a professional or an expert. The rhythm and tempo are consistent throughout the entire duration of the clip.

LTU (Gong et al., 2023)

The skill level of the tap dancer cannot be inferred from the audio clip.

Pengi (Deshmukh et al., 2024)

It is a big crowd.

SALMONN (Tang et al., 2023)

The skill level of the tap dancer can be inferred from the nature and arrangement of sounds. If the sounds are complex and intricate, it suggests that the dancer is highly skilled. On the other hand, if the sounds are simple and repetitive, it suggests that the dancer may not be as skilled.

Ques.: Given the variety of sounds including mechanisms, generic impact sounds, and water splashes, determine the most probable activity occurring during the audio clip. Consider the presence of baby laughter in your analysis.

Ques.: Inferring from the nature and arrangement of sounds, what can be said about the skill level of the tap dancer?

66 of 69

66

GAMA: Dense Captioning Examples

GAMA (Ghosh et al., 2024)

The man is speaking, followed by a beep and then breathing sounds, indicating a pause or break in his speech or activity before continuing to speak again after a brief moment of silence or restoration.

Qwen (Bai et al., 2023)

A man speaks and beeps occur.

Pengi (Deshmukh et al., 2023)

A man speaks and a woman speaks.

SALMONN (Tang et al., 2023)

This is a recording of a man speaking with a beeping sound in the background. The man is speaking in a calm and measured tone. The beeping sound is consistent and repetitive throughout the recording.

GAMA (Ghosh et al., 2024)

​

The sound is a cricket chirping, which is a common nighttime insect noise often associated with peaceful outdoor environments like forests or gardens at dusk when crickets are most active and vocalize to attract mates.

Qwen (Bai et al., 2023)

Birds chirping in the forest.

Pengi (Deshmukh et al., 2023)

Birds are chirping and crickets are chirping.

SALMONN (Tang et al., 2023)

This is a recording of a bird. The bird is chirping loudly and consistently throughout the entire recording. The bird's chirps are high-pitched and rapid, almost sounding like a tweet. The bird's chirping is the only sound in the recording.

67 of 69

67

GAMA: Dense Captioning Examples

GAMA (Ghosh et al., 2024)

The man is speaking while a duck quacks, suggesting a casual outdoor setting or a pet store environment where animals are present and humans interact with them for entertainment or care purposes.

Qwen (Bai et al., 2023)

A man speaks while a machine runs and scrapes.

Pengi (Deshmukh et al., 2023)

A man speaking and a duck quacking.

SALMONN (Tang et al., 2023)

The sound of a pig squealing and grunting can be heard. This is followed by the sound of a man's voice speaking. The man's voice can be heard over the sound of the pig squealing and grunting. The pig's squealing and grunting can be heard in the background of the man's voice.

GAMA (Ghosh et al., 2024)

The sound is a toilet flushing, with some background noise and surface contact.

Qwen (Bai et al., 2023)

Clanking and tapping followed by a flushing sound.

Pengi (Deshmukh et al., 2023)

A door is opened and a door is opened.

SALMONN (Tang et al., 2023)

A toilet is flushed, releasing a gush of water into the bowl. The sound of the flush is loud and echoes in the small bathroom. The toilet gurgles as the water drains away.

68 of 69

68

GAMA: Complex QA Examples

GAMA (Ghost et al., 2024)

The truck is probably a diesel, as indicated by the heavy engine sounds and the absence of a typical gasoline engine's characteristic high-pitched whine at idle.

LTU (Gong et al., 2023)

Based on the sound of acceleration and revving, it is likely that the engine type is a high-performance or sports car engine.

Pengi (Deshmukh et al., 2024)

It is diesel.

SALMONN (Tang et al., 2023)

The audio contains the sound of an engine revving and accelerating. Based on this, it is likely that the engine type is a high-performance gasoline engine, such as a V8 or a turbocharged four-cylinder. The sound of the engine revving and accelerating is a common characteristic of high-performance engines, which are designed to produce a lot of power and torque. The sound of the engine can vary depending on the specific make and model of the vehicle, but the general characteristics of a high-performance engine can be identified by the sound of the engine revving and accelerating.

GAMA (Ghost et al., 2024)

The man is likely giving instructions to passengers or announcing a station stop. He may also be explaining the train's route or destination.

LTU (Gong et al., 2023)

The man may be giving a lecture or presentation in the subway car while people are moving around him, possibly taking notes.

Pengi (Deshmukh et al., 2024)

It is a man.

SALMONN (Tang et al., 2023)

It is difficult to determine the exact actions the man might be performing based solely on the combination of subway sounds and male speech. However, it is possible that the man is giving a public announcement or warning to the passengers on the subway.

Ques.: Analyze the audio and infer the likely engine type based on the sound of acceleration and revving.

Ques.: Based on the combination of subway sounds and male speech, suggest the actions the man might be performing at this time.�

69 of 69

69

AF2: Overall Architecture

Audio Flamingo 2 is based on the cross-attention architecture, where we condition our custom CLAP. The model is trained on 3 stages of training and can perceive audios up to 5 minutes in length.