Advancing multimodal emotion analysis: integrating machine learning and deep learning approaches

Yunyi Zhu

doi:10.1117/12.3052226

11 December 2024 Advancing multimodal emotion analysis: integrating machine learning and deep learning approaches

Yunyi Zhu

Proceedings Volume 13445, International Conference on Electronics, Electrical and Information Engineering (ICEEIE 2024); 134452P (2024) https://doi.org/10.1117/12.3052226
Event: International Conference on Electronics. Electrical and Information Engineering (ICEEIE 2024), 2024, Haikou, China

Abstract

Multimodal emotion analysis, blending machine learning and deep learning, is transforming computer-based human emotion recognition. This review examines the complexity of human emotions, generally classified into six types: joy, sadness, anger, fear, surprise, and disgust. It covers both unimodal and multimodal techniques, emphasizing their utility in fields like medicine, driving safety, and law enforcement. Unimodal technologies analyze single data types (text, image, audio), while multimodal fusion algorithms integrate these for heightened accuracy. Traditional machine learning methods, such as SVM and k-NN, are foundational for feature extraction and classification. Deep learning, using CNN and RNN architectures, advances emotion recognition by extracting intricate data patterns. Combining traditional machine learning with deep learning enhances precision, with the CNN-LSVM approach notably boosting classification accuracy. The paper focuses on three primary multimodal techniques: feature-level, decision-level, and hybrid fusion. Feature-level fusion merges various modality features to enrich classifier inputs. Decision-level fusion consolidates outcomes from different modalities, ensuring stable classification. Hybrid fusion, merging the prior two, offers a holistic approach for accurate emotion analysis. These methods, however, face challenges like increased computational demands and complex model training.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Yunyi Zhu "Advancing multimodal emotion analysis: integrating machine learning and deep learning approaches", Proc. SPIE 13445, International Conference on Electronics, Electrical and Information Engineering (ICEEIE 2024), 134452P (11 December 2024); https://doi.org/10.1117/12.3052226

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available