1 of 15

Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data

Sreyan Ghosh¹², Sonal Kumar², Zhifeng Kong¹, Rafael Valle¹,

Bryan Catanzaro¹, Dinesh Manocha²

¹Nvidia, USA

²University of Maryland, College Park, USA

1

2 of 15

Limited Audio Datasets: Audio data is scarce compared to other modalities, with minimal availability for certain tasks. Existing dataset are small and many datasets are restricted or lack real-world diversity. This proves to be a problem in learning good audio understanding models.

2

The Problem of Data in Audio Processing

�

Non-verbal Audio Datasets

Largest Dataset - ~1k Hours

Vision Datasets

Largest Dataset - 5B+ I-T pairs

Language Datasets

Largest Dataset - Billions of tokens

Speech Datasets

Largest Dataset - 60k+ Hours

3 of 15

3

Introduction

Existing data augmentation methods (noise augmentation or masking) do not capture the true diversity present in real-world audios.

Synthetic data enables the creation of tailored training datasets, mitigating the impact of noise in web datasets and serving as a valuable resource in low-resource scenarios.

The use of synthetic data to improve understanding and reasoning is booming in language and vision (e.g., Phi4, GPT-4o).

There is lack of foundational audio generation models. Existing models lack the ability to precisely control generation.

We propose Synthio, a novel synthetic data generation technique to generate diverse and consistent synthetic audios to improve audio classification.

Traditional augmentation, such as SpecAug, degrades performance on small-scale datasets. Naive augmentation outperforms traditional methods significantly but plateaus with higher sample counts. Synthio further enhances performance by generating consistent and diverse synthetic data.

4 of 15

4

Synthio: Motivation

Problem 1 – Alignment: When prompted with “sound of a bus” for the category bus in the TUT-Urban dataset, the generated audio may not reflect the typical bus sounds in European cities (where TUT was recorded), as bus sounds can vary by region, with some featuring loud engines and dense crowds while others have quieter engines and sparse crowds.

Can we just fine-tune the text-to-audio model? -- Fine-tuning on small datasets lead to overfitting and generation of audios that sound similar.

Sound of a bus from the TUT Urban dataset.

Sound of a bus from a text-to-audio model trained on AudioSet.

Problem 2 – Lack of feature diversity: When prompted with “Sound of a park’”, we observed that 9 out of 10 times, the model generated the sound of children playing as part of the generated audio.

Two sounds with different random seeds, CFG and inference steps.

5 of 15

5

Synthio: Aligning the Text-to-Audio Model using Preference Optimization

Step 1: Constructing the Preference Dataset

Generate template-based captions for each instance (e.g., "Sound of a [label]").
Prompt the T2A model multiple times per instance, pairing generated audio with ground-truth audio, treating generated as "loser" and ground-truth as "winner.”

Step 2: Preference Optimization with DPO (Rafailov et. al)

Train the T2A model on the preference dataset using DPO.
The optimized model improves alignment with ground-truth audio.

We align the T2A model with the small-scale dataset using DPO, ensuring generated audio matches its acoustic characteristics.

6 of 15

6