How I built a complete speech emotion recognition pipeline from audio preprocessing to real-time inference—published at IEEE DECoN 2025. Covers the published lightweight CNN approach and the extended project implementation with CNN-BiLSTM.

Published — IEEE DECoN 2025
DOI: 10.1109/DECoN67170.2025.11448083
Title: Speech Emotion Recognition: A Human-Centric Framework with Enhanced Data Augmentation and Lightweight CNN

Speech Emotion Recognition (SER) is a critical component of affective computing—enabling machines to understand not just what is said, but how it's said. This project presented the full engineering challenge: from raw audio to real-time emotion classification.

Two Architectures: Published vs. Extended

This project comprises two related but distinct implementations:

1. Published Research (IEEE DECoN 2025)

Lightweight CNN + Enhanced Augmentation — optimized for the paper's scope:

Lightweight CNN architecture (no recurrent layers)
Enhanced data augmentation pipeline (pitch shifting, time stretching, noise injection, SpecAugment)
Speaker-independent evaluation on RAVDESS and CREMA-D
Sub-100ms inference latency

2. Extended Project Implementation

Hybrid CNN + BiLSTM + Attention — broader engineering exploration:

CNN for local spectral pattern extraction
Bidirectional LSTM for temporal dependency modeling
Attention pooling for weighted temporal feature aggregation
Real-time streaming inference pipeline

Both share the same preprocessing foundation. The extended implementation builds on the published work to explore temporal modeling and real-time deployment.

The Problem

Human emotional expression in speech varies enormously:

Speaker variability: Pitch, accent, speaking rate differ wildly
Recording conditions: Background noise, microphone quality, room acoustics
Class imbalance: Some emotions (neutral) dominate datasets
Real-time constraints: Sub-100ms inference for conversational AI

Shared Preprocessing Pipeline

Audio loading with librosa (consistent sample rate: 22050 Hz)
Silence removal using energy-based VAD
Signal normalization to [-1, 1] range
Fixed-length segmentation (3-second windows with overlap)

Published Architecture: Lightweight CNN

The IEEE DECoN 2025 paper focuses on a lightweight CNN with enhanced augmentation:

CNN layers: Local spectral pattern extraction from MFCC sequences
No recurrent layers — purely convolutional for speed
Enhanced augmentation: Pitch shifting, time stretching, noise injection, SpecAugment
Output: 7-8 emotion classes (neutral, happy, sad, angry, fear, disgust, surprise)

Extended Architecture: Hybrid CNN + BiLSTM + Attention

The broader project implementation explores temporal modeling:

CNN layers: Local spectral pattern extraction from MFCC sequences
BiLSTM layers: Temporal dependency modeling across utterance
Attention pooling: Weighted aggregation of temporal features
Output: 7-8 emotion classes (neutral, happy, sad, angry, fear, disgust, surprise)

Key Engineering Challenges

1. Speaker Generalization

Models easily overfit to training speakers. Solutions:

Data augmentation: Pitch shifting, time stretching, noise injection
Speaker-independent splits: Train/test by speaker, not sample
Domain adversarial training: Gradient reversal for speaker invariance

2. Class Imbalance

Neutral class often 40%+ of data. Used:

Weighted cross-entropy with inverse frequency weights
Focal loss to focus on hard examples
Oversampling minority classes in each batch

3. Real-Time Inference

ONNX export for optimized inference
Quantization (INT8) for edge deployment
Streaming chunk processing with overlap-add for continuous audio

Results

Published Work (IEEE DECoN 2025)

The paper reports competitive accuracy with the lightweight CNN on speaker-independent splits. See the publication for detailed metrics.

Extended Implementation (Development Measurements)

Disclaimer: The following are development measurements from the extended CNN-BiLSTM implementation during project testing, not controlled benchmark results with statistical validation. They reflect the broader project's experimental configuration and should not be cited as formal benchmarks.

Dataset	Classes	Accuracy	F1 (Macro)	Latency
RAVDESS	8	~78%	~0.75	45ms
CREMA-D	6	~73%	~0.70	52ms

Conditions: Speaker-independent splits, single-sample inference, warm model, CPU/GPU varies.

Lessons Learned

Preprocessing > Architecture: Clean, consistent audio pipeline mattered more than model tweaks
MFCCs still work: Despite raw waveform trends, MFCCs provide strong perceptual features with low compute
Train/test split by speaker: Random splits inflate metrics; speaker-independent is the real test
Consistency is deployment: Same preprocessing at train and inference time is non-negotiable
Hybrid CNN-RNN > Pure approaches: Local patterns + temporal context both matter for emotion
Lightweight can be enough: The published CNN achieved competitive results without recurrent layers—simpler is often better

Future Work

Transformer-based speech models (Wav2Vec2, HuBERT fine-tuning)
Self-supervised speech representation learning
Multimodal emotion recognition (audio + video + text)
Cross-lingual emotion recognition
Noise-robust speech processing with denoising front-ends
Edge deployment (TensorRT, CoreML, TFLite)

Related Project

This blog post accompanies the Speech Emotion Recognition project case study.

View Project Case Study →

Resources

Publication

Speech Emotion Recognition: A Human-Centric Framework with Enhanced Data Augmentation and Lightweight CNN
IEEE DECoN 2025 (Distributed Computing and Networking Conference)
DOI: 10.1109/DECoN67170.2025.11448083
Year: 2025

Key Contributions:

Enhanced data augmentation pipeline (pitch shifting, time stretching, noise injection, SpecAugment) tailored for SER
Lightweight CNN architecture achieving competitive accuracy with sub-100ms inference latency
Speaker-independent evaluation on RAVDESS and CREMA-D datasets
Analysis of augmentation impact on cross-speaker generalization

Building Real-Time Speech Emotion Recognition