Marathwada Mitra Mandal's�College of Engineering�Karvenagar, Pune 52 

​

​

Accredited with "A" Grade by NAAC

Recipient of "Best College Award" in AY 2018-19 from SPPU

​

Department of Information Technology

​

Deshmukh Ankita

Kalluray Abhijeet

Pacharne Rutuja

Sambhudas Akash

Guide

Prof. Nikhil Dhavase

Introduction

• Voice interaction with computer and machine is one of the evolving fields in Human Computer Interaction (HCI)
• Speech Processing is used for various things including voice interaction (Voice Interface)
• Understanding what the emotions of the user are during such a process is one of the important aspects of voice processing.
• This is termed a Speech Emotion Recognition (SER)

Human Computer

Interaction

Speech

Processing

Speech Emotion Recognition

Literature Review

• This project focus on using acoustic features for SER, and not the content (directly) in the speech.
• Hence different papers were studied which provided techniques to extract the features from an audio signal.
• Following are the papers and the feature extraction technique selected.
1. Mel Frequency Cepstral Coefficients for Music Modeling -> MFCC

[http://ismir2000.ismir.net/papers/logan_paper.pdf]

• Chroma Feature Extraction -> Chroma Spectral Features

[https://www.academia.edu/42216949/Chroma_Feature_Extraction]

• Speech Emotion Recognition from 3DLog-Mel Spectrograms -> Mel Spectrogram
• Chroma Feature Extraction -> Chroma
• Detecting Harmonic Change In Musical Audio -> Tonnetz
• Music type classification by spectral contrast feature -> Spectral Contrast
2. “A Speech Emotion Recognition Model Based on Multi-Level Local Binary and Local Ternary Patterns” this paper was also studied which uses the above mentioned feature extractions, and also uses a unique way in the preprocessing stage of the audio signals.

​

​

​

​

​

​

​

​

Problem Statement / Abstract

• Problem Statement: Identifying the underlying emotion in human speech using acoustic features and comparing the different featuring extraction technique.
• The emotion recognition is based on the acoustic features of the sound and not the words spoken in the speech.
• No linguistic processing is carried out in the entire process.
• With different emotions the tone, intensity, frequency, pitch of the human speech changes. These changes can be studied to predict the underlying emotion.
• We also answer the question if multiple features work better than their individual counterparts.

Scope, Objective(s)

AIM/OBJECTIVE

• Proper processing of sound waves to capture the acoustic features present in it.
• Analyzing and classifying the speech into its sentimental category based upon acoustic features only.
• Comparing the accuracy of the model with different feature extraction technique.

​

​

The different domains of HCI where SER is used are:

• Call centre applications
• Assistive telecommunication
• Psychosis monitoring
• Diagnosis of neuropsychological disorders
• Audio Mining

​

System Architecture

• The ML model works in the following steps:
• Loading the audio data
• Extracting features from the audio files
• Model training using the features
• Predicting emotion for the given audio file

​

​

​

Load Data

Feature Extraction

Model Training

Prediction

Project Design

Gather known data with know labels

Prepare data (preprocess)

Choose Algorithm and tune parameters

Use model with new data for prediction

Admin

User

System

Methodology

1. Loading Data
• The audio data used for this project was a combination of 2 datasets The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) and CORPUS dataset
• These datasets were chosen as they are opensource and free to use.
• The RAVDESS dataset contains both video and audio data out of which only the audio data was used.
• The audio data contains different categorizes of emotions, out of which 4 emotions were selected to train the model ( Angry, Sad, Neutral, Happy)
• These were the emotions selected as these emotions instance were more in number compared to others.

​

Methodology

The distribution of dataset is all follows:

​

angry 324

happy 324

sad 324

neutral 252

Methodology

2. Extracting Features
• MFCC (Mel Frequency Cepstral Coefficients)

MEL scale

• We Humans don’t perceive loudness on a linear scale (to double the loundness around 8 times more energy is used)
• The Mel scale relates perceived frequency, or pitch, of a pure tone to its actual measured frequency.

​

Cepstral

• Cepstral is the word Spectral with the ‘spec’ part reversed.
• Cepstrum is the information of rate of change in spectral bands

Methodology

2. Extracting Features
• MFCC (Mel Frequency Cepstral Coefficients)

20 – 40 ms

Mel Filter

Sum

Methodology

2. Extracting Features
• Spectral Contrast
• Similar approach as MFCC, but rather than using the Mel scale, it uses the peaks and the valleys of the signal. In general peaks the strong spectral peaks roughly correspond with harmonic components; while non-harmonic components, or noises, often appear at spectral valleys.
• Thus spectral contrast better reflects the relative distribution of the harmonic and non-harmonic components in the spectrum.

20 – 40 ms

Octave Scale Filter

Peak/Valley Select and

Spectral Contrast

Spectral Contrast Coefficient

Methodology

2. Extracting Features
• Chroma
• The chroma feature is a descriptor, which represents the tonal content of a musical audio signal in a condensed form.
• The underlying observation is that humans perceive two musical pitches as similar in color if they differ by an octave
• The twelve pitch spelling attributes as used in Western music notation

{C, C♯, D, D♯, E , F, F♯, G, G♯, A, A♯, B}

0

1

2

3

4

5

6

7

8

9

10

11

Chroma Vector

Methodology

2. Extracting Features
• Chroma
• Given a music representation (e.g. a musical score or an audio recording), the main idea of chroma features is to aggregate for a given local time window (e.g. specified in beats or in seconds) all information that relates to a given chroma into a single coefficient.
• Shifting the time window across the music representation results in a sequence of chroma features each expressing how the representation's pitch content within the time window is spread over the twelve chroma bands
• The resulting timechroma representation is also referred to as chromagram.
• The figure given, shows the chromograms for C-major scale.

​

1. Musical score of a C-major scale.
2. (b) Chromagram obtained from the score.
3. (c) Audio recording of the C-major scale played on a piano.
4. (d) Chromagram obtained from the audio recording

Methodology

2. Extracting Features
• Mel Spectrogram Frequency

​

Mel Scale

• We know Mel scale now.
• The scale that plots the actual thing to what we humans perceive.

Spectrogram

• A spectrogram is a visual representation of the spectrum of frequencies of a signal as it varies with time

Sound Recording

Fourier Transform

Visualizing (but nothing can be seen)

Using Mel scale to covert the Hz to Mel

Spectrogram

Log scaling

Methodology

2. Extracting Features
• Tonnetz
• In musical tuning and harmony, the Tonnetz (German: tone-network) is a conceptual lattice diagram representing tonal space first described by Leonhard Euler in 1739.
• Various visual representations of the Tonnetz can be used to show traditional harmonic relationships in European classical music.

​

Methodology

2. Extracting Features
• Tonnetz
• This model maps 12-bin chroma vectors to the interior space of a 6-D polytope; pitch classes are mapped onto the vertices of this polytope musical tuning and harmony.
• 6-dimensional basis representing the perfect fifth, minor third, and major third each as two-dimensional coordinates.

Methodology

Comparison Procedure

• A comparison is carried out with different permutation of following things.
• 2 datasets
• 5 feature extraction techniques
• 2 selection criteria

​

The train, test split was carried at 80% and 20% respectively with stratified splitting

Methodology

Model Training

• A Convolution Neural Network was used as a model for prediction
• A convolution neural network was used to identify patterns in the data.
• They are mostly used in images, but a 1 dimensional Convolution can be used for this project
• The following is the convolution architecture with 17 layers.
• The model has an accuracy of 92% accuracy on training set and 82% accuracy on the testing set.

• The architecture of the CNN is as show
• The first phase consists of 3 convolution layers with size 256, 512, 256 respectively
• The end phase consists of Dense layers with size 128, 64, 4, 4
• Layers like pooling, activation, flatten, dropout are also added whenever they are necessary

Results

• The table shows the comparison of individual features.
• The ‘mel’ and ‘mfcc’ features have the highest accuracy.
• This is true for both the CORPUS JL and RAVDESS dataset

Results

• The table shows the comparison of a combination of multiple features.
• Here we can see that the model with CORPUS JL dataset, and all the features mel, mfcc, and contrast performs the best out of all, with a combined accuracy of 0.9128

Results

4. Prediction
1. Here is the classification report of the best model on the test set.

​

Result

4. Prediction
1. Here is the confusion matrix shown in the image
2. An ideal confusion matrix would be a diagonal one, where the diagonal elements are light(high value) and other elements are darker(low value)
3. As seen in the confusion matrix the model sometimes cannot predict correctly between sad and neutral. This can be a cause as both sad and neutral emotion have a lower tone resulting in weak features.

​

​

Predicted Values

Actual Values

Implementation

• Programming Language used: Python
• Libraries used:
• NumPy – (for numerical analysis)
• Librosa – (for audio and music analysis)
• Tensorflow – (for building the model)
• Scikit –Learn – (for scoring of the model)
• Seaborn, Matplotlib – (for plotting the graphs)
• Os, pickle, glob - (for file handling)
• PyQt5 – (for GUI Designing)

​

Implementation

Implementation

Implementation

• This is the GUI of the application.
• It allows the user to either record the audio or upload a pre-existing file
• The graphs of different features are shown in the black window
• The highest prediction probability class is displayed on the right along with its probability

Conclusion

• Using just the intensity and the variance of the sound waves in a human speech, the underlying emotions can be understood without linguistic processing.
• This means that acoustic features are just as important factor to be studied in SER with its big brother NLP.
• Even without knowing the actual words the sound waves can be analyzed to understand the emotion, this result can further be developed by classifying the words in the speech based on their sentiments, positive and negative. This combination would yield much more better results.
• MFCC still stands to be the best feature extraction technique and a combination of all the features does not provide a significant improvement over MFCC alone.

References

1. The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS)

[https://zenodo.org/record/1188976]

• The dummy’s guide to MFCC

[https://medium.com/prathena/the-dummys-guide-to-mfcc-aceab2450fd#:~:text=Mel%20scale%20is%20a%20scale,in%20speech%20at%20lower%20frequencies).]

• Mel Frequency Cepstral Coefficient (MFCC) tutorial

[http://practicalcryptography.com/miscellaneous/machine-learning/guide-mel-frequency-cepstral-coefficients-mfccs/

• Mel Frequency Cepstral Coefficients for Music Modeling

[http://ismir2000.ismir.net/papers/logan_paper.pdf]

• Chroma Feature

[https://en.wikipedia.org/wiki/Chroma_feature#:~:text=Chroma%2Dbased%20features%2C%20which%20are,to%20the%20equal%2Dtempered%20scale.]

• Chroma Feature Extraction

[https://www.academia.edu/42216949/Chroma_Feature_Extraction]

• Mel Spectogram

[https://towardsdatascience.com/getting-to-know-the-mel-spectrogram-31bca3e2d9d0#:~:text=The%20Mel%20Scale%2C%20mathematically%20speaking,in%20distance%20from%20one%20another.]

• Music type classification by spectral contrast feature

[https://www.researchgate.net/publication/3968978_Music_type_classification_by_spectral_contrast_feature]

• Tonnetz

[https://en.wikipedia.org/wiki/Tonnetz]

• Detecting Harmonic Change In Musical Audio

[http://rose.ofai.at/~martin.gasser/papers/oefai-tr-2006-13.pdf]

​