Improving the accuracy of speech emotion recognition based on ensemble classification with adaptive fuzzy aggregation using the Gompertz function

Abstract

The paper addresses the problem of speech emotion recognition and investigates an approach to improving the efficiency of ensemble classification based on adaptive fuzzy aggregation. A method is proposed for aggregating predictions from machine learning and deep learning classifiers using the Gompertz function, in which the number of the most informative classes is determined adaptively for each object based on the normalized entropy of the probability distribution. Unlike the standard approach with a fixed number of the most informative classes, the proposed method takes into account the level of uncertainty of a specific prediction, which allows for dynamically changing the number of classes involved in further aggregation. An experimental study was conducted on the Ryerson Audio-Visual Database of Emotional Speech and Song dataset. Eight base classifiers, each operating on different representations of audio data, were used to construct a heterogeneous ensemble. For tabular spectral features, the Backward Feature Elimination procedure was additionally applied, which allowed to form individual feature spaces for the base classifiers. As part of the experiment, two hundred and nineteen ensemble classifiers were formed and evaluated with each aggregation method. Standard fuzzy aggregation based on the Gompertz function with a fixed number of the most informative classes achieved an accuracy of eighty-one point three nine percent and an F1-score of eighty-one point two zero percent, whereas the proposed adaptive method achieved an accuracy of eighty-five point five six percent and an F1-score of eighty-five point five five percent. Thus, the adaptive method resulted in a four point one seven percent increase in accuracy and a four point three five percent increase in the F1-score compared to standard fuzzy aggregation.