1 of 13

EAB Meeting • 17 June 2022

Differentiable signal processing �for audio engineering

Christian J. Steinmetz

c.j.steinmetz@qmul.ac.uk

Centre for Digital Music, Queen Mary University of London

Huy Phan

Joshua D. Reiss

2 of 13

More people are creating audio content

Music

Podcasts

Short-form content

Sound for Video

3 of 13

Producing high quality audio requires expertise

Demand for high quality audio

4 of 13

Deep learning for audio processing

Neural network

Source separation

Speech enhancement

Audio effect modeling

Stöter et al., 2019, "Open-unmix-a reference implementation for music source separation." JOSS

Pascual et al., 2017 "SEGAN: Speech enhancement generative adversarial network." arXiv:1703.09452

Martínez Ramírez et al., 2020, "Deep learning for black-box modeling of audio effects." Applied Sciences

5 of 13

Audio engineers solve problems with DSP

Controlling audio effects

Modeling acoustic spaces

Creating a mix

Can we build models that learn to control DSP for audio engineering tasks?

6 of 13

Differentiable signal processing

Neural network

Signal processing

  • Leveraging existing DSP tools and knowledge
  • High quality audio processing with few artifacts
  • Human understandable outputs that can be adjusted
  • Efficient and can easily run in real-time on CPU

Control parameters

7 of 13

Techniques

  1. Automatic differentiation (AD)�Engel et al. 2020

​

  • Neural proxies and hybrids (NP)�Steinmetz et al. 2020, Steinmetz et al. 2022

​

  • Numerical gradient approximation (NGA)�Martinez Ramirez et al. 2021

​

​

8 of 13

Colonel and Steinmetz et al., 2022 "Direct design of biquad filter cascades with deep learning by sampling random polynomials." IEEE ICASSP

Steinmetz et al., 2021 "Filtered noise shaping for time domain room impulse response estimation from reverberant speech." IEEE WASPAA (Best Student Paper Award)

Steinmetz et al., 2022 "Style transfer of audio effects with differentiable signal processing." Journal of the Audio Engineering Society

Differentiable IIR filters

Differentiable reverberation

Generalized differentiable effects

9 of 13

What’s next?

Torchdiffx

Differentiable audio effects in PyTorch

​

  • Gain
  • Panner
  • Lowpass/Highpass
  • Parametric EQ
  • Dynamic range compressor

​

Coming soon

​

  • Multiband compressor
  • Brickwall limiter
  • Denoiser
  • Artificial reverberation
  • Overdrive/Distortion

Applications in...

​

  • Automatic multitrack mixing
  • Deconstructing a mix
  • Audio quality enhancement

Improved methods to facilitate backprop through DSP

10 of 13

EAB Meeting • 17 June 2022

Differentiable signal processing �for audio engineering

Christian J. Steinmetz

c.j.steinmetz@qmul.ac.uk

Centre for Digital Music, Queen Mary University of London

Huy Phan

Joshua D. Reiss

11 of 13

Differentiable signal processing

Deep neural network

Signal processing

Control parameters

Make this differentiable

12 of 13

Deep learning for audio processing

Deep neural network

Great for problems that can’t be solved with DSP

but...

    • Prone to artifacts or adding signal distortions
    • Not easily controllable (how do we adjust the output?)
    • Requires significant computation (often not real-time on CPU)

13 of 13

Differentiable signal processing

Backprop through DSP requires special techniques