Diffusion-based generative models for audio

GLADIA Research Group

Speakers:  Michele Mancusi 

Giorgio Mariani   

Generative Models

Generative Models

Generative Models: Diffusion models

• Diffusion models define a sequence of steps to slowly add random noise to data (forward process) and then learn to reverse it (backward process).
• This allows to obtain probable data points from random noise.

Understanding Diffusion Models: Langevin Dynamics

Brownian motion

Pollen grains in water

Simulation of particles

moving in water

https://water.lsbu.ac.uk/water/Brownian.html

Langevin Dynamics

For very tiny particles (few microns):

​

Brownian motion

Brownian motion

Brownian motion

Fokker-Planck equation

Steady State of Langevin Dynamics

if we want to sample from the distribution  we need to set the potential energy to be

Our equation becomes

Sampling Using Langevin Dynamics

We can use the Euler-Maruyama method

Sampling Using Langevin Dynamics

Illustration from https://yang-song.net/blog/2021/score

Diffusion Models

Score function

Score function!

Score-based model

We need to train a model to estimate the score function

​

by minimizing this loss (a.k.a. Fisher divergence)

​

The ideal workflow…

Two important issues

1^st Problem: Low-density region

How can we bypass the difficulty of accurate score estimation in regions of low data density? 

Solution: add noise to the data

How much noise?

• Larger noise can cover more low density regions for better score estimation, but it over-corrupts the data.
• Smaller noise causes less corruption of data, but does not cover the low density regions.

​

Solution: Multiple Levels!

Annealed Langevin dynamics combine a sequence of Langevin chains with gradually decreasing noise scales.

Annealed Langevin Dynamics

2^nd Problem: Unknown data score

Solution: Denoising Score Matching

Noise Conditional Score Networks (NCSN)

The Network

Forward and Backward Diffusion

Forward Diffusion Process

Backward Diffusion Process

Backward Diffusion Process

(Song et al., 2019; Ho et al. 2020; Song et al., 2021)

What about audio?

• Audio is lagging behind:
• Limited availability of data and copyright issues
• Non standardized benchmarks for comparisons
• Recycle ideas from vision instead of new things
• Cannot embed audio on paper
• Peculiarities of the audio domain:
• Multiple sources in the same mix, compositionality problem
• Audio has very long contexts (GPT4 max context is 25,000 tokens, a second of high-quality audio is 44,100 samples)

​

Focus on music – Commercial interest

Data Representation

Symbolic

Continuous

Data Representation

• Waveform domain:
• 1D signal
• Sampling rate:
• High resolution: 44.1 kHz (CD format)
• In deep learning often use 16 kHz
• Bit depth:
• 16-bit: 2^16 (or 65,536)
• 24-bit: 2^24 (or 16,777,216)
• Common formats:
• WAV (Waveform Audio File Format):
• No compression
• MP3 (MPEG-1 Audio Layer III):
• Compression, bitrate (# of bits transmitted per second) range of 32-320 kbps

​

Multi-Source Diffusion Models

Core idea

Multi-Source Diffusion Models

Inference procedure: Total Generation

Inference procedure: Partial Generation

Inference procedure: Source Separation

Quantitative metrics

Future work

• Introduce text-conditioning for fine-grained control
• Latent-space inference
• Train over more realistic data by bootstrapping SOTA regressors (Demucs)
• Train symbolic version on MIDI data (which is more abundant)

Thank You!