1 of 50

DeepVQE: from research to every Teams call

Evgenii Indenbom, IC3 AI, Microsoft

DeepVQE: from research to every Teams call

1

Evgenii Indenbom, Microsoft

2 of 50

IC AI Europe

DeepVQE: from research to every Teams call

2

Evgenii Indenbom, Microsoft

3 of 50

About us

  • IC3 AI Europe:�We focus on audio quality improvement models and developing ML video codec
  • IC3 AI: AI research team�The team focuses on improving audio and video quality using AI and providing AI based features
  • IC3 - Intelligent conversation and communication cloud
    • Platform for communication: audio and video calling, meetings, chat
    • Users: Teams, Skype, Azure Communication Services, Xbox, etc.
    • Microsoft Development Center Estonia mostly focused on these areas
  • Microsoft�Our mission statement is “to empower every person and every organization on the planet to achieve more.”

​

  • Me: ML Scientist at IC3 AI

DeepVQE: from research to every Teams call

3

Evgenii Indenbom, Microsoft

4 of 50

What is VQE?

  • VQE stands for Voice Quality Enhancer
  • DeepVQE – NN-model for voice quality enhancement in real-time communication

​

  • Real-time communication ⬄ the same as phone call?:
    • At least 2 parties
    • Each party has a device with microphone and speakers
    • Capture microphone signal and play it back on the other party speakers

​

  • Why do we need to enhance voice quality?

DeepVQE: from research to every Teams call

4

Evgenii Indenbom, Microsoft

5 of 50

Acoustic echo

  • Uncancelled echo is very annoying

​

  • Seems easy to remove, but it’s not

​

  • It might be hard to interrupt your partner:�This is called doubletalk scenario�
    • Microphone
    • DSP-based echo canceler

​

DeepVQE: from research to every Teams call

5

Evgenii Indenbom, Microsoft

network

6 of 50

Audio call

DeepVQE: from research to every Teams call

6

Evgenii Indenbom, Microsoft

Room impulse response (RIR)

Room

(Lossy) network

(Lossy) network

Far-end

Near-end

Noise

Mic-in signal

Mic-out signal

Far-end signal

7 of 50

Quality degradation sources

  • Local:
    • Acoustic echo
    • Ambient noise
    • Reverberation
    • Microphone distortion

​

  • Network/RTC related:
    • Packet loss
    • Network jitter
    • Network/processing delay

DeepVQE: from research to every Teams call

7

Evgenii Indenbom, Microsoft

8 of 50

DeepVQE

  • DeepVQE is voice quality enhancement NN-model:
    • Far-end echo cancellation
    • Noise suppression
    • Dereverberation

​

  • DeepVQE is deployed in MS Teams:
    • More than 200 million daily users
    • 6 platforms (Windows, Mac, Linux, Android, iOS and WebASM)
    • x64 and ARM CPU architectures

​

  • DeepVQE is the best in the class solution for real-time communication

DeepVQE: from research to every Teams call

8

Evgenii Indenbom, Microsoft

9 of 50

Demo: Noise suppression

DeepVQE: from research to every Teams call

9

Evgenii Indenbom, Microsoft

Microphone

DSP-based noise suppression

AI Model

10 of 50

Demo: AEC – doubletalk scenario

DeepVQE: from research to every Teams call

10

Evgenii Indenbom, Microsoft

Far end

Microphone

AI Model

11 of 50

Demo: Dereverberation

DeepVQE: from research to every Teams call

11

Evgenii Indenbom, Microsoft

Model

Microphone

12 of 50

DeepVQE: constraints and challenges

  • Constraints:
    • Algorithmic latency 20ms
    • Processing time P99 < 7ms per 10ms frame
    • Needs to run in-client: on mobile devices, on 10+ year old CPUs etc.
  • Challenges:
    • 100+ languages, including tonal etc.
    • 1000+ audio device manufactures
    • Multiple device types (headsets, Bluetooth, conference equipment)
    • Different room types and sizes
  • Needs to be very robust and accurate: audio domain is unforgiving
    • To guarantee 99.9% reliability for a call with 10 participants, each client must have 99.99% reliability rating

​

DeepVQE: from research to every Teams call

12

Evgenii Indenbom, Microsoft

13 of 50

Prior art (2020)

  • DSP-based approach:
    • Backbone – adaptive filters
    • Does not work well with:
      • Non-stationary noise
      • Echo path changes
      • Non-linear distortions
      • Double talk

​

  • TFNet – noise suppression NN-model:
    • Deployed in MS Teams

​

  • Recent research papers demonstrated feasibility of NN-based residual echo cancellers

DeepVQE: from research to every Teams call

13

Evgenii Indenbom, Microsoft

14 of 50

Step 1: build the model

  • Test data
  • Test metrics
  • Train data
  • Loss function
  • Network architecture

DeepVQE: from research to every Teams call

14

Evgenii Indenbom, Microsoft

15 of 50

Real-world data

  • Use Amazon Mturk for real devices and rooms
  • Standalone app (Windows, Android) that can capture both mic and loopback signal
  • Capture multiple scenarios
    • Near-end single talk, (audio from user)
    • Non-speech utterances (uhmmm, laughter etc)
    • Far-end single talk (audio from PC speakers)
    • Far-end single talk with echo path change �(PC/user moves)
    • Double talk (audio from PC speakers and user)
    • Double talk with echo path change
    • Sweep
  • Currently 100k recordings�from ~8000 rooms/devices

DeepVQE: from research to every Teams call

15

Evgenii Indenbom, Microsoft

16 of 50

Testing audio quality

  • Simple objective metrics has low correlation with perceptual quality:
    • ERLE - Echo Return Loss Enhancement
    • Mean square error (MSE): in time, frequency or MEL domain
    • PESQ - Perceptual Evaluation of Speech Quality
  • Subjective evaluation (Mean Opinion Score):
    • Requires specialized framework to ensure uniform evaluation
    • Rater qualification tests (gold questions, trapping questions)
    • Ensuring high correlation between expert and crowd sourced ratings
    • Repeatability testing etc.
  • IC3 AI implemented 2 subjective test frameworks:
    • ITU-R P.835: noise suppression quality evaluation (part of TFNet project)
    • ITU-R P.831: echo canceller quality evaluation (done by IC3 AI Tallinn)

​

DeepVQE: from research to every Teams call

16

Evgenii Indenbom, Microsoft

17 of 50

DNSMOS835 and AECMOS

​

DeepVQE: from research to every Teams call

17

Evgenii Indenbom, Microsoft

18 of 50

Training data

  • We need triplets of audio clips:
    • Far-end signal
    • Microphone signal
    • Target signal
  • Hard/impossible to obtain:
    • Requires complex studio setup
    • Will not match real distribution
    • Many loss functions require target and output to be synched in phase: impossible to achieve with multiple microphones
  • Solution: synthesizer

DeepVQE: from research to every Teams call

18

Evgenii Indenbom, Microsoft

19 of 50

Synthesizer

DeepVQE: from research to every Teams call

19

Evgenii Indenbom, Microsoft

Noise

User speech

Farend

Echo

Microphone

Clean/target

neural network

20 of 50

What’s inside the synthesizer?

  • Datasets:
    • Clean speech for target signal:
      • Filtered dns_librivox, common_voice, podcasts, singing, emotional etc.
    • Noisy/clean speech for far end signal:
      • dns_librivox, common_voice, podcasts, singing, emotional etc.
    • Noise:
      • dns_challenge and others
    • Room impulse responses:
      • RWCP, Bar-Ilan Acoustic Lab, Aachen AIR, simulated etc.
    • Non-linear distortions:
      • Trained on real data
  • Most datasets can be found on challenges GitHub:

​

  • Signal mixing
  • Augmentations:
    • Volume
    • Microphone distortions
    • Bandpass filtering
    • Spectral shaping
    • Jitter simulation
  • Scenario transitions:
    • Double talk
    • Near end single talk
    • Far end single talk

DeepVQE: from research to every Teams call

20

Evgenii Indenbom, Microsoft

21 of 50

Towards model: AEC challenges

  • Microsoft has organized multiple challenges:
    • AEC Challenges at InterSpeech 2021, ICASSP 2021, 2022, 2023
    • 5 DNS Challenges, 2 PLC Challenges and 2 SIG Challenges
  • Each AEC challenge is 15-30 participant teams:
    • Amazon, Alibaba, Baidu, ByteDance, Kuaishou Samsung, Tencent, etc.
    • Chinese Academy of Science and many universities
  • Challenge result gives us:
    • Insights into the current state of the art
    • Broad search in model architecture space
    • Training data for objective metrics

​

DeepVQE: from research to every Teams call

21

Evgenii Indenbom, Microsoft

22 of 50

Time vs frequency domain

  • Models can work in:
    • time domain (waveform)
    • frequency domain (spectrogram)

​

  • In practice, frequency domain models:
    • easier to implement
    • have better run time performance
    • have much in common with image processing, allowing to borrow ideas

DeepVQE: from research to every Teams call

22

Evgenii Indenbom, Microsoft

STFT

Short-time Fourier transform

But it’s non-trivial to model phase:

23 of 50

CRUSE�Convolutional Recurrent U-net for Speech Enhancement

  • Proposed by Sebastian Braun (MSR)
  • Adapted for echo cancellation:
    • Stacked microphone and far end spectra
    • Microphone and far end are pre-aligned with DSP algorithm
  • First production candidate model

DeepVQE: from research to every Teams call

23

Evgenii Indenbom, Microsoft

24 of 50

Loss function

  • AECMOS – could not use it. One DNN easily cheats another.
  • Complex compressed MSE loss in frequency domain:
    • Power law compression factor 0.3
    • Use other window size than model STFT (1024 samples)
    • And smaller hop size (256 samples)
  • GAN loss:
    • Widely used, but gives contradictory results
    • Tends to cheat objective metrics (AECMOS, DNSMOS)
    • From small to negligible impact on human ratings

DeepVQE: from research to every Teams call

24

Evgenii Indenbom, Microsoft

25 of 50

Step 2: deploying the model

  • Inference engine:
    • C/C++ interface
    • 1 inference per 10ms
    • Must run on low-end CPU
  • MS Teams client (MSRTC) integration:
    • 5 major operating systems
    • 4 CPU architectures
  • MSRTC provided us with:
    • Client telemetry
    • A/B Test framework

DeepVQE: from research to every Teams call

25

Evgenii Indenbom, Microsoft

26 of 50

ONNX Runtime (ORT)

  • General purpose inference engine
  • Benchmark:
    • Back-to-back processing
    • Intel Core i7 10700K @ 3.8GHz
    • Average frame time: 0.53ms

​

  • Telemetry:
    • Average frame time: > 1.25ms
    • Frames exceeding 5ms: > 1%
    • Frames exceeding 10ms: > 0.01%

DeepVQE: from research to every Teams call

26

Evgenii Indenbom, Microsoft

27 of 50

Why numbers are so different?

  • CPU frequency (Lenovo ThinkPad P15s Gen 1):
    • Nominal: 2.8 GHz
    • Turbo Boost: 4.2 GHz
    • Minimal: 0.4 GHz
  • Maximum throttling – 10x
  • Reasons for throttling:
    • Power limits: the bigger integrated GPU load => the lower CPU frequency
    • Depleted battery and power saving
    • Thermal limit/overheating

DeepVQE: from research to every Teams call

27

Evgenii Indenbom, Microsoft

28 of 50

Other hardware related factors

  • CPU generation:
    • IPC has increased substantially over last 6 generations
    • Number of ALU/vector units
  • CPU cache sizes:
    • Memory bandwidth to L1/L2/L3/DRAMM drops drastically with cache level
    • Mobile/Pentium/Celeron processors has significantly smaller cache sizes
  • Memory:
    • Memory bandwidth is proportional to the number of memory channels
    • DDR-3 is a way slower than DDR-4
  • Context switches:
    • Back-to-back inference allows to reuse caches
    • 1 frame per 10ms means caches are flushed meanwhile

​

​

DeepVQE: from research to every Teams call

28

Evgenii Indenbom, Microsoft

29 of 50

What slows ORT down?

  • Memory allocations:
    • Create overhead (up to 50% in ORT 1.7)
    • The main cause for latency spikes:�Page allocation requires kernel call => spike in latency
  • State transfer:
    • Convolution causal padding is state�and needs to be copied over
  • Poor vectorization in some cases:
    • Batch size: 1
    • Sequence size: 1

​

DeepVQE: from research to every Teams call

29

Evgenii Indenbom, Microsoft

memory allocations

state transfer

30 of 50

Conv2d �state transfer

DeepVQE: from research to every Teams call

30

Evgenii Indenbom, Microsoft

31 of 50

Can ORT be faster?

  • Performance optimizations attempts:
    • Quantization
    • Smaller models
    • Improvements in ORT codebase:
      • 30% improvement of Conv2d performance
  • Multiple threads:
    • Increased latency spikes (due to thread priority mismatch)
    • Busy looping second core 95% of time

​

  • Still too slow:
    • Too complex, too general codebase:
      • General purpose engine requires complex interfaces and nested abstractions
    • Non-typical use case: small frame size, many inferences:
      • Overhead (negligible in other tasks) taking over 30% of time

​

DeepVQE: from research to every Teams call

31

Evgenii Indenbom, Microsoft

Version

Frame Time

1.7

0.53ms

1.9

0.46ms

1.11

0.37ms

32 of 50

Frame Processing Inference Engine (FPIE)

  • Designed for frame processing:
    • Batch size is 1
    • Sequence size is 1
    • Channels last memory layout
  • Built-in circular buffer states
  • Modularized design with 2 abstraction layers:
    • Vectorized math engine (AVX, NEON)
    • Network graph and layers
  • Runs many DNN architectures:
    • TFNet, CRUSE, DeepVQE, E3Net, DeepPLC etc.
    • TFNet was also deployed with custom inference, but it can run TFNet architecture

DeepVQE: from research to every Teams call

32

Evgenii Indenbom, Microsoft

33 of 50

Making FPIE run … faster

  • Logbook:
    • 2w: FPU/compiler naïve implementation: 50% slower than ORT
    • 1m: Naïve AVX vectorization: 30% faster than ORT
    • 8m: FPIE is 4-5 times faster than ORT

​

  • How could it be?
    • SIMD – single instruction, multiple data
    • CPU has more than one (2-4) ALU
    • CPU and ALU have pipeline, multiple instructions running in parallel
    • Code structure must allow CPU to fill the instruction pipeline
    • Memory is a bottleneck: read once, use multiple times

​

​

DeepVQE: from research to every Teams call

33

Evgenii Indenbom, Microsoft

34 of 50

Optimizing matrix multiplication

Optimization strategies:

    • Increasing block size by using register pairs (or bigger register blocks)
    • Batching rows and summing odd and even (and tertiary) rows independently

DeepVQE: from research to every Teams call

34

Evgenii Indenbom, Microsoft

Block Size

Batch Size

Frame Time

8

1

0.299ms

8

2

0.206ms

16

1

0.211ms

8

4

0.179ms

16

2

0.167ms

16

4

0.148ms

ORT-1.11

​

1.28ms

35 of 50

FPIE and multithreading

  • Single-threaded A/B test (higher complexity model):
    • 10ms frame time ~0.5ms
    • Frames over 5ms ~0.028%
  • Double-threaded A/B test:
    • 10ms frame time ~0.3ms
    • Frames over 5ms ~0.019%

​

  • Hybrid thread-synchronization model:
    • Busy-loop synchronization within frame inference
    • Lock-free message queue: 1 message per frame

​

​

​

DeepVQE: from research to every Teams call

35

Evgenii Indenbom, Microsoft

36 of 50

Modeling challenges and edge cases

  • Far end signal misalignment leading to echo leaks.

DSP-based algorithm poorly handles:

      • Variable echo delay (mainly varying audio driver processing time)
      • Distorted echo signal (speaker or microphone distortion)
  • Slow scenario switches

Model adapts slowly during:

      • NEST to FEST transitions leaking echo
      • FEST to NEST transitions suppressing near end speech
      • 0.5s suppression means 4 lost words
  • Edge cases:
    • Tonal languages, dialects
    • Non-speech audio (laughter, uhmms, coughing etc.)
    • Poor or unusual devices: bandpass, clipping, distortion etc.

​

DeepVQE: from research to every Teams call

36

Evgenii Indenbom, Microsoft

37 of 50

Alignment block

  • Far end and microphone features maps are aligned using local cross-attention
  • Soft-weighted far end feature map replaces hard decision of DSP-delay estimator

​

  • Completely eliminated echo leakage reports
  • The first model deployed on Windows for all users

DeepVQE: from research to every Teams call

37

Evgenii Indenbom, Microsoft

38 of 50

Alignment block V2

  • Second generation alignment:
    • Multi-channel similarity
    • 2d-convolution to smoothen and aggregate similarity channels

​

  • Substantial improvement in double talk metrics due to improved alignment

DeepVQE: from research to every Teams call

38

Evgenii Indenbom, Microsoft

39 of 50

Complex convolving mask

  • Traditional magnitude mask:
    • Uses single value per frequency bin
    • Binary decision: suppress or pass
  • Complex mask:
    • Allows to correct phase
    • Requires 3 channels encoding to stabilize training and inference
    • Provides little or no improvement over magnitude mask
  • Complex convolving mask:
    • Decoder produces convolution kernel instead of single value for each frequency bin
    • Convolution is causal in time dimension
    • Production model uses 2x3 mask

DeepVQE: from research to every Teams call

39

Evgenii Indenbom, Microsoft

mask

microphone spectrum

enhanced spectrum

40 of 50

DeepVQE architecture

Additional improvements

    • Residual convolution block
    • Sub-pixel convolution decoder

DeepVQE: from research to every Teams call

40

Evgenii Indenbom, Microsoft

41 of 50

Training pipeline optimizations

  • Synthesizer improvements:
    • Scenario transitions
    • Additional augmentations (bandpass etc.)
    • New datasets (common voice, podcasts etc.)
  • Hyper parameters:
    • Weight decay
    • Loss weight
  • RNN chunking

DeepVQE: from research to every Teams call

41

Evgenii Indenbom, Microsoft

42 of 50

ICASSP 2023 challenge results

DeepVQE: from research to every Teams call

42

Evgenii Indenbom, Microsoft

ICASSP 2023 Noise Suppression Grand Challenge

​

ICASSP 2023 Echo Cancellation Grand Challenge

​

43 of 50

DeepVQE: production model

  • 480 000 parameters
  • Average frame time ~0.25 (Windows, all devices)
  • Deployed on Windows, Mac

DeepVQE: from research to every Teams call

43

Evgenii Indenbom, Microsoft

44 of 50

Training pipeline

  • Azure ML 8x NVIDIA V100 32Gb GPUs
  • Training time 20-40 hours (production size model)

DeepVQE: from research to every Teams call

44

Evgenii Indenbom, Microsoft

45 of 50

Android

  • ARM NEON vectorized math engine
    • Apple Mac Mini M1 faster than Intel Core i7 10700K
  • Model time per 10ms frame 0.3ms (Samsung A52)
  • But number of frames exceeding 5ms or even 10ms is very high
  • Why?
  • ARM big.LITTLE:
    • Cortex-A76 (big): 0.3ms
    • Cortex-A55 (LITTLE): 2ms
  • CPU throttling from 2 GHz to 600 MHz => 7ms per 10ms frame

DeepVQE: from research to every Teams call

45

Evgenii Indenbom, Microsoft

46 of 50

FP16

  • IEEE-754 floating point 16-bit data type
  • Supported by most modern ARM devices
    • But with notable exceptions: RedMi Note 11 and Co
  • ARM big.LITTLE:
    • Cortex-A76 (big): 0.18ms
    • Cortex-A55 (LITTLE): 0.81ms
  • CPU throttling from 2 GHz to 600 MHz => 2.5ms
  • Improvements are due to:
    • Double SIMD vector size
    • Twice less CPU cache usage => model better fits into LITTLE core cache

​

​

DeepVQE: from research to every Teams call

46

Evgenii Indenbom, Microsoft

47 of 50

Android: no real-time scheduler

  • Model still fallbacks to high percentage of over 10ms frames
  • Time slice on Android is 4ms
    • Bi-modal processing time distribution
  • Real-time priority is unavailable for regular applications
  • Double-threaded inference is inefficient:
    • High chance of double stop
    • => 8ms or even bigger delay

​

DeepVQE: from research to every Teams call

47

Evgenii Indenbom, Microsoft

VQE processing time (20ms frame)�from real call

48 of 50

iOS

  • Disabling built-in echo canceller leads to device AGC failure on newest iPhones
  • The model is deployed in noise suppression mode

DeepVQE: from research to every Teams call

48

Evgenii Indenbom, Microsoft

49 of 50

What’s more?

  • Packet loss concealment (deployed in 2023)
  • Personalized noise suppression (deployed in 2024)
  • Automatic passive enrollment (in internal testing)
  • Signal enhancement:
    • Bandwidth expansion (especially important for PSTN and Bluetooth devices)
    • Declipping
    • etc.
  • Bring Your Own Device (BYOD) – revolution in hybrid meetings

DeepVQE: from research to every Teams call

49

Evgenii Indenbom, Microsoft

  • FPIE and DeepVQE are integrated:
    • Teams web-client
    • Xbox and Windows
  • Improving user experience with auxiliary outputs:
    • Music detection
    • Secondary speaker detection
    • Noise detection
  • Optimizations:
    • Model size decreased to 340K parameters
    • FPIE performance improved by 10%

50 of 50

DeepVQE: from research to every Teams call�Evgenii Indenbom, IC3 AI, Microsoft

DeepVQE: from research to every Teams call

50

Evgenii Indenbom, Microsoft

Questions?