Publication detail

MixedEmotions: An Open-Source Toolbox for Multimodal Emotion Analysis

BUITELAAR, P. WOOD, I. NEGI, S. ARCAN, M. MCCRAE, J. ABELE, A. ROBIN, C. ANDRYUSHECHKIN, V. ZIAD, H. SAGHA, H. SCHMITT, M. SCHULLER, B. SÁNCHEZ-RADA, J. IGLESIAS, C. NAVARRO, C. GIEFER, A. HEISE, N. MASUCCI, V. DANZA, F. CATERINO, C. SMRŽ, P. HRADIŠ, M. POVOLNÝ, F. KLIMEŠ, M. MATĚJKA, P. TUMMARELLO, G.

Original Title

MixedEmotions: An Open-Source Toolbox for Multimodal Emotion Analysis

Type

journal article in Web of Science

Language

English

Original Abstract

Recently, there is an increasing tendency to embed functionalities for recognizing emotions from user-generated media content in automated systems such as call-centre operations, recommendations, and assistive technologies, providing richer and more informative user and content profiles. However, to date, adding these functionalities was a tedious, costly, and time-consuming effort, requiring identification and integration of diverse tools with diverse interfaces as required by the use case at hand. The MixedEmotions Toolbox leverages the need for such functionalities by providing tools for text, audio, video, and linked data processing within an easily integrable plug-and-play platform. These functionalities include: 1) for text processing: emotion and sentiment recognition; 2) for audio processing: emotion, age, and gender recognition; 3) for video processing: face detection and tracking, emotion recognition, facial landmark localization, head pose estimation, face alignment, and body pose estimation; and 4) for linked data: knowledge graph integration. Moreover, the MixedEmotions Toolbox is open-source and free. In this paper, we present this toolbox in the context of the existing landscape, and provide a range of detailed benchmarks on standard test-beds showing its state-of-the-art performance. Furthermore, three real-world use cases show its effectiveness, namely, emotion-driven smart TV, call center monitoring, and brand reputation analysis.

Keywords

emotion analysis, open source toolbox, affective computing, linked data, audio processing, text processing, video processing

Authors

BUITELAAR, P.; WOOD, I.; NEGI, S.; ARCAN, M.; MCCRAE, J.; ABELE, A.; ROBIN, C.; ANDRYUSHECHKIN, V.; ZIAD, H.; SAGHA, H.; SCHMITT, M.; SCHULLER, B.; SÁNCHEZ-RADA, J.; IGLESIAS, C.; NAVARRO, C.; GIEFER, A.; HEISE, N.; MASUCCI, V.; DANZA, F.; CATERINO, C.; SMRŽ, P.; HRADIŠ, M.; POVOLNÝ, F.; KLIMEŠ, M.; MATĚJKA, P.; TUMMARELLO, G.

Released

23. 8. 2018

ISBN

1520-9210

Periodical

IEEE TRANSACTIONS ON MULTIMEDIA

Year of study

20

Number

9

State

United States of America

Pages from

2454

Pages to

2465

Pages count

12

URL

https://ieeexplore.ieee.org/document/8269329/?arnumber=8269329

BibTex

@article{BUT155798,
  author="BUITELAAR, P. and WOOD, I. and NEGI, S. and ARCAN, M. and MCCRAE, J. and ABELE, A. and ROBIN, C. and ANDRYUSHECHKIN, V. and ZIAD, H. and SAGHA, H. and SCHMITT, M. and SCHULLER, B. and SÁNCHEZ-RADA, J. and IGLESIAS, C. and NAVARRO, C. and GIEFER, A. and HEISE, N. and MASUCCI, V. and DANZA, F. and CATERINO, C. and SMRŽ, P. and HRADIŠ, M. and POVOLNÝ, F. and KLIMEŠ, M. and MATĚJKA, P. and TUMMARELLO, G.",
  title="MixedEmotions: An Open-Source Toolbox for Multimodal Emotion Analysis",
  journal="IEEE TRANSACTIONS ON MULTIMEDIA",
  year="2018",
  volume="20",
  number="9",
  pages="2454--2465",
  doi="10.1109/TMM.2018.2798287",
  issn="1520-9210",
  url="https://ieeexplore.ieee.org/document/8269329/?arnumber=8269329"
}