Skip to content

Read-only version: For security reasons, the Research Information System (FIS) has been temporarily replaced by this read-only version. All public content and full texts remain available; login, editing, submission and statistics are not available. Questions: fis@uni-bamberg.de

Home › Publications › Towards Robust and Interpretable Practical Applications of Automatic Mental State Analysis Using a Dynamic and Hybrid Facial Action Estimation Approach

Publication

Towards Robust and Interpretable Practical Applications of Automatic Mental State Analysis Using a Dynamic and Hybrid Facial Action Estimation Approach

Hassan, Teena (2020): Towards Robust and Interpretable Practical Applications of Automatic Mental State Analysis Using a Dynamic and Hybrid Facial Action Estimation Approach, Bamberg: Otto-Friedrich-Universität, doi: 10.20378/irb-48641.

Details

Faculty/Chair
Author
Hassan, Teena
Publisher Information
Otto-Friedrich-Universität : Bamberg
Year of publication
2020
Pages
xxviii, 159, Illustrationen, Diagramme
Supervisor
Beyerer, Jürgen
Language
English
Remark

Kumulative Dissertation, Otto-Friedrich-Universität Bamberg, 2020

DOI
Licence
URN
Abstract

This dissertation presents a probabilistic state estimation framework for integrating data-driven machine learning models and a deformable facial shape model in order to estimate continuous-valued intensities of 22 different facial muscle movements, known as Action Units (AU), defined in the Facial Action Coding System (FACS). A practical approach is proposed and validated for integrating class-wise probability scores from machine learning models within a Gaussian state estimation framework. Furthermore, driven mass-spring-damper models are applied for modelling the dynamics of facial muscle movements. Both facial shape and appearance information are used for estimating AU intensities, making it a hybrid approach. Several features are designed and explored to help the probabilistic framework to deal with multiple challenges involved in automatic AU detection. The proposed AU intensity estimation method and its features are evaluated quantitatively and qualitatively using three different datasets containing either spontaneous or acted facial expressions with AU annotations. The proposed method produced temporally smoother estimates that facilitate a fine-grained analysis of facial expressions. It also performed reasonably well, even though it simultaneously estimates intensities of 22 AUs, some of which are subtle in expression or resemble each other closely. The estimated AU intensities tended to the lower range of values, and were often accompanied by a small delay in onset. This shows that the proposed method is conservative. In order to further improve performance, state-of-the-art machine learning approaches for AU detection could be integrated within the proposed probabilistic AU intensity estimation framework.

In addition to AU intensity estimation, this dissertation explores the applicability of the estimated AU intensities for automatic analysis of mental states such as pain and distraction. A survey of automatic pain detection approaches, conducted as part of this dissertation, highlights the progress and deficits in this field. Several AU-based rules are designed for pain intensity estimation based on psychological evidence, and their performance is evaluated empirically. The potential of these AU-based rules to automatically generate explanations for pain detections is also illustrated. Furthermore, a preliminary analysis of the estimated AU intensities shows differences in facial actions between various distraction scenarios during simulated driving.

The results of this dissertation show that more interdisciplinary research is needed to address the open challenges in the field of automatic mental state analysis, particularly to build reference datasets, to model interpersonal differences, and to generate human-comprehensible explanations of predictions.

GND Keywords
Mimik
FACS
Gesichtserkennung
Maschinelles Lernen
Mensch-Maschine-Kommunikation
Keywords
Facial expression analysis, facial action units, pain detection, state estimation, machine learning, interpretability, information fusion, mental state analysis, distraction detection
DDC Classification
004 Computer science
RVK Classification
ST 300
Type
Doctoralthesis
URI
Activation date
2020-10-29
Awards

Permalink: https://fis.uni-bamberg.de/handle/uniba/48641 · Data as JSON · Cite: BibTeX · RIS

Data as of: 2026-11-16

Restricted read-only access: login, editing and submission are temporarily disabled.