Sound Event Localization and Detection

of Overlapping Sources in 3D

Using Convolutional Recurrent Neural Networks

Sharath Adavanne

Tampere University of Technology, Finland

Overview

• Introduction
• Definitions
• Motivation
• Previous works
• Method
• Features
• Neural network
• Training procedure
• Evaluation
• Datasets
• Baselines
• Metrics
• Results
• Conclusion

​

​

Sound Event Classification / Audio Tagging

Sound Event Detection

Sound Event Detection

Sound Event Localization

Sound Event Localization and Detection (SELD)

Applications

SELD automatically describes social and human activities

• Assist hearing impaired to visualize sounds
• Robots/humanoids can use SELD for navigation and natural interaction with surroundings
• Military, smart cities, industries and smart homes could use it for surveillance
• Smart meeting rooms - recognize speech and its direction, use it for beamforming and enhance teleconferencing or ASR
• Bio-diversity monitoring
• Assist users to visualize sounds in VR

Sound event localization and detection

Can be broadly divided into two sub-tasks

1. Sound event detection
2. Sound source localization

Sound event detection

• Identify the temporal onset and offset of individual sound event, recognize the sound class
• Sound events are overlapping and polyphonic in nature - ‘Polyphonic sound event detection’
• Very active research domain
• DCASE - Detection and classification of acoustic scenes and events http://dcase.community
• Annual IEEE Audio and Acoustic Signal Processing (AASP) challenge
• 2013, 2016 (Budapest), 2017 (Munich), 2018 (Surrey)

​

​

DCASE 2016: SED Synthetic data

DCASE 2017: SED Synthetic data

DCASE 2016: SED Real-life data

DCASE 2017: SED Real-life data

Sound event detection (SED)

• Tackled as a supervised multi-label multi-class classification task
• Common classifiers: GMM-HMM, SVM, FC, RNN, CNN, CRNN
• Current state of the art from DCASE 2017 - CRNN: Convolutional recurrent neural network

​

Multichannel SED

• To improve recognition of overlapping sound events
• Achieved top results in DCASE 2016 and 2017

​

​

​

​

​

​

Multichannel SED

• Recently, we studied SED performance on identical sound scene with single, binaural and first-order Ambisoincs (FOA) microphones [1].
• The results showed that overlapping sound events can be recognized better with higher number of channels (FOA).

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

[1] Sharath Adavanne, Archontis Politis and Tuomas Virtanen "Multichannel sound event detection using 3D convolutional neural networks for learning inter-channel features" in International Joint Conference on Neural Networks (IJCNN 2018)

Sound source localization

• Task of determining the direction or position of a sound source with respect to the microphone
• We only deal with the estimation of sound event direction, referred as direction-of-arrival (DOA) estimation.
• DOA methods in literature can be categorized broadly into parametric and deep neural network (DNN)-based

​

DOA estimation: Parametric methods

• Popular parametric methods - time difference of arrival (TDOA), steered response power (SRP), multiple signal classification (MUSIC) and estimation of signal parameters via rotational invariance technique (ESPIRIT)
• These methods vary in algorithmic complexity, constraints in array geometry, and model assumptions on acoustic scenarios.
• Subspace method MUSIC are generic to array and can produce high resolution DOA estimates of multiple sources.
• Cons
• Requires good estimate of number of active sources
• Found to be sensitive to reverberant and low SNR scenarios

DOA estimation: DNN-based methods

​

• To overcome some of the drawbacks of parametric methods
• Implementing localization task in DNN framework allows seamless integration into higher level learning based end-to-end sound analysis and detection systems. Eg: for SELD and human-robot interaction.
• Existing DOA based methods have only been implemented and studied on static and point sources, and have been shown to perform equally or better than parametric methods in reverberant scenarios.
• These methods can potentially learn the number of active sources from the data itself.

Summary of DNN-based DOA estimation methods

• Input features: method- and array- specific
• Classification vs regression:
• Classification - estimate likelihood of a fixed set of angles,
• Regression - continuous high resolution DOA estimates
• Output format: full or partial azimuth, elevation and distance estimation; x, y, and z estimation
• Number of sources: single or overlapping
• Array structures: Linear, binaural, circular, cartesian, robot/humanoid, distributed

​

SELD: Data association problem

• When SED and DOA estimation of multiple overlapping sound events are done separately.
• Which sound event has what location?
• What if number of sound events detected by SED and DOA estimators are different?
• One of the solution: Jointly predict SED and DOA.
• Hirvonen et al. [1] is the only existing DNN-based method. CNN maps audio features from circular array to eight angles in full azimuth for each sound event class in dataset
• Other works combined SED and parametric DOA estimation
• Lopatka et al. [2]: non-overlapping sound events, MPEG-7 features, SVM classifier
• Butko et al. [3]: distributed microphone arrays, overlapping two at a time, GMM-HMM for SED, SRP for DOA estimation. Data association problem!
• Chakraborty et al.[4]: Replaced SRP in [3] with sound-model based DOA estimation to solve data association problem.

​

[1] T. Hirvonen, “ Classification of spatial audio location and content using convolutional neural networks”, in Audio Engineering Society Convention 138, 2015

[2] K. Lopatka, J. Kotus and A. Czyzewsk, “Detection, classification and localization of acoustic events in the presence of background noise for acoustic surveillance of hazardous situations”, in Multimedia Tools and Applications Journal, vol. 75, no. 17, 2016

[3] T. Butko, F. G. Pla et al, “Two-source acoustic event detection an dlocalization: Online implementation in a smart-room”, in European Signal Processing Conference, 2011

[4] R. Chakraborty and C. Nadeu, “Sound-model-based acoustic source localization using distributed mircophone arrays”, in International Conference on Acoustics, Speech and Signal Processing, 2014.

​

​

SELD as a classification task

Hirvonen et al. [1] extracted the frame-wise spectral power from each microphone of circular-array and used a CNN classifier to map it to eight angles in full azimuth for each sound event class in the dataset.

​

Cons:

• The resolution of DOA is limited to trained directions, and performance on unseen labels is not known.
• For larger datasets with higher number of sound event and increased resolution along azimuth and elevation directions, this results in enormous number of output nodes. Training such a DNN where the number of positive classes per frame is one or two with respect to a high number of negative class poses challenges of an imbalanced dataset.
• Additionally, it required a huge dataset with enough examples to train for each individual class-DOA pair.

Pro:

• Allows simultaneous recognition of more than one instance of the same sound event in a given time frame.

​

[1] T. Hirvonen, “ Classification of spatial audio location and content using convolutional neural networks”, in Audio Engineering Society Convention 138, 2015

Proposed method: SELDnet

• Jointly estimate SED and DOA using a DNN
• Use magnitude and phase component of spectrogram as input, generic to array configurations. To prove this we show results on circular-array and Ambisonic format.
• We use the state-of-the-art multichannel and polyphonic SED method using CRNN that won DCASE 2017 real-life SED task and extend it to estimate the respective DOAs for the recognized sound events in regression manner.

​

SELDnet

​

• We tie every sound event output obtained in classification mode with three outputs (x, y and z) in regression mode.
• By using SED output as confidence measure for choosing the respective DOAs we extend SOTA polyphonic SED performance for polyphonic SELD.
• We show that network learns better by predicting x, y and z coordinates of DOA instead of azimuth/elevation angels.
• By estimating DOA in regression manner, we enable the network to estimate DOA in continuous space and show that the method works seamlessly on unseen DOA labels

SELDnet: SED output

• Groundtruth output: 0 - inactive, 1 - active
• Sigmoid activation for classification layer,
• Network output: [0, 1], threshold with 0.5
• enables multi-class multi-label classification.
• Binary cross-entropy loss between groundtruth and network output

​

​

SELDnet: DOA output

• A 3D Cartesian DOA estimate on a unit sphere centered at origin are in the x, y, z = [-1, 1] range
• Groundtruth output: x, y, z = 0, 0, 0 - inactive, active - reference DOA in [0 1] range.
• Tanh activation for regression layer,
• Network output: [-1, 1]
• multi-class multi-output regression.
• MSE loss between groundtruth and network output.
• By using MSE we truly represent the distance between two points in space.

​

​

SELDnet training

• Weighted combination of MSE and binary cross-entropy loss
• 1000 epochs with early stopping if SELD score on test split did not improve for 100 epochs
• Adam optimizer
• Keras library with Tensorflow backend

​

Available online! https://github.com/sharathadavanne/seld-net

Datasets

Evaluated on seven datasets, with stationary and point sources

​

​

​

​

​

Each dataset has three subsets with maximum one/two/three temporally overlapping sound events. Each of this subset has a three cross-validation splits.

​

Baselines

Compared with six baselines on both 2D and 3D DOA estimation

Metric: SED

• Polyphonic SED metrics: F-score and error rate (ER) calculated in segments of one seconds.
• Ideal SED method has F-score of one and error rate of zero

Metric: DOA estimation

​

• Evaluated using frame-wise DOA error and frame recall
• DOA error: the central angle between the estimated and ground truth DOA

​

​

​

• Frame recall: To account for frames where the number of estimated and groundtruth DOAs are unequal.

Frame recall = TP / (TP+FN)

• Ideal SELD method will have DOA error of zero and frame recall of 100%

​

​

​

​

​

Metric: SELD score

• During training of SELDnet, we perform early stopping based on the combined SELD score.
• An ideal SELD method will have an SELD score of zero.

​

​

Results: Network architecture

Results: FFT size and sequence length

Results: output weights

Results: DOA output format

Results: visualization of SELDnet output

for overlapping sound events

Results: Continuous DOA estimation and performance on unseen DOA labels

Results: Ambisonic, anechoic/reverberant datasets

• SED metrics of SELDnet are comparable if not better than baseline MSEDnet
• SELDnet is significantly better than baseline DOAnet in terms of frame recall. Improvement in frame recall is a direct results of using SED output as a confidence measure, thereby extending SOTA SED performance to SELD.
• DOA estimation is consistently better than baseline MUSIC which uses the number of sources knowledge
• Unmatched reverberant dataset: SELDnet is robust to reverberation and the model trained on a single room (Room 1) can be used for similarly sized room configurations (Room 2 and 3)

Results: Ambisonic, anechoic/reverberant datasets

Results: Ambisonic Real-life IR datasets

• Performance drops with real data in comparison to synthesized data. With larger dataset REALBIG the SELDnet performance is seen to improve.
• In presence of Ambiance network performance is robust with 10 and 20 dB SNR, and gave comparable results to REALBIG. Performance drops at 0 dB SNR.

Results: Circular-array, anechoic/reverberant datasets

• Generic to array structure.
• SELDnet-azi, baselines HIRnet and AZInet estimates DOA in azimuth only.
• SELDnet has significantly better frame recall than the best baseline AZInet.
• Between SELDnet and SELDnet-azi, even though frame recall is in the same order the DOA error of SELDnet-azi is better. Showing that estimation of DOA in 3D is challenging using circular array.

​

Results: moving sound sources, no overlap

​

Results: moving sound sources, max two overlap

Conclusion

• Proposed a CRNN based method SELDnet to simultaneously recognize, localize and track sound events with respect to time.
• Localization is done by estimation DOA on a unit sphere around the microphone using 3D Cartesian coordinates.
• We show that using regression helps estimate DOA in a continuous space, and also estimate unseen DOA labels seamlessly.
• The usage of SED output as confidence measure to estimate DOA was seen to extend the SOTA SED performance to SELD.

​

Conclusion

• The proposed method uses phase and magnitude spectrogram as input feature, thereby making the method generic and easily extendable to different array structures. This was proven by evaluation on Ambisonic and circular-array datasets.
• In comparison to parametric and DNN based baselines, the proposed SELDnet is shown to be robust to reverberation, low SNR scenarios and unseen rooms with comparable room sizes.
• Overall performance on dataset synthesized using real-life IR was seen to drop in comparison to artificial IR data, suggesting the need for a larger real-life training datasets and more powerful classifiers in future.

Thank you

Sharath Adavanne

MLS recording : https://youtu.be/8vplZkuLMWs

Ambiance recording: https://youtu.be/BxmrEAqwKok

​

​