PainFaceReader – Automatic pain detection

Objective

Development of a monitoring system for automatic pain detection using action units.
A special focus in pain research is on reliable pain detection based on changing facial expressions as a form of non-verbal communication and how to clearly differentiate them from those produced by other emotional states. Here we use our SHORE^® emotion analysis software and the Facial Action Coding System (FACS) to reliably detect action units.

In collaboration with the University of Bamberg, our goal is to create an autonomous system that can automatically detect pain in patients who are unable to communicate – and do so in a timely manner and when medical staff are not present.

Partners

Chair of Applied Computer Sciences / Cognitive Systems at the University of Bamberg

• Using machine learning approaches to interpret preprocessed action units

Chair of Physiological Psychology at the University of Bamberg

• Database for machine learning
• Evaluation of results

Fraunhofer IIS | Facial Analysis Solutions

• Emotion recognition in a variety of environmental, lighting and perspective contexts, based on reliable detection and identification of action units with the help of machine learning

The project is being funded by the Deutsche Forschungsgemeinschaft DFG (project number 405630557) since 2018.

The unambiguous classification of facial expressions using action units plays a major role in pain research. People with cognitive impairments that leave them unable to articulate their pain, such as those suffering from dementia or undergoing acute post-operative care, have only their facial expressions to communicate with others.

The challenge

Using such a system to help make diagnoses and decide when to prescribe pain medication faces two challenges: one, how to reliably identify a person’s state of mind; and two, how to explain classification decisions. Facial responses tied to annoyance, anger, disgust and pain exhibit similar action units or even the same action units appearing in a different order. The action units detected point directly to a particular state of mind.
The proposed system is to function autonomously so that medical staff are called only in an emergency, which means more targeted use of personnel while reducing costs.
 

To achieve a high degree of sensitivity as well as specificity in the classification of pain, we are working on a way to reliably detect action units that takes into account combinations of those units while integrating time-variant models.

• Co-occurrence of action units delivers important additional information
• The characteristics related to the time sequence of action units provide another source of information
  

We are also working to further develop our SHORE^® software library in terms of valid detection of concurrent action units and consideration of time sequences. To this end, we apply machine learning methods with a focus on computer vision (deep neural networks).
 

• Intensive care
• Palliative medicine
• Facial expression analysis in depression
   

• Jaspar Pahl, Ines Rieger, Dominik Seuß; presented at the first Workshop on the Art of Learning with Missing Values (Artemiss) hosted by the 37th International Conference on Machine Learning (ICML) 2020: Multi-label Learning with Missing Values using Combined Facial Action Unit Datasets

• Ines Rieger, Jaspar Pahl, Dominik Seuß; accepted at the 15th IEEE International Conference on Automatic Face and Gesture Recognition 2020, Workshop "Affect Recognition in-the-wild: Uni/Multi-Modal Analysis & VA-AU-Expression Challenges": Unique Class Group Based Multi-Label Balancing Optimizer for Action Unit Detection

• Jaspar Pahl, Ines Rieger, Dominik Seuß; this submission is subject to the Action Unit detection task of the Affective Behavior Analysis in-the-wild (ABAW) challenge at the IEEE Conference on Face and Gesture Recognition 2020: Multi-Label Class Balancing Algorithm for Action Unit Detection

• Blais, C.; Fiset, D.; Furumoto-Deshaies, H.; Kunz, M.; Seuss, D.; Cormier, S.; Journal of Pain, 2019: Facial Features Underlying the Decoding of Pain Expressions
  
• Mahesh, B.; Prassler, E.; Hassan, T.; Garbas, J.U.; International Conference on Pervasive Computing and Communications (PerCom) 17, 2019: Requirements for a Reference Dataset for Multimodal Human Stress Detection

• Miriam Kunz, Dominik Seuss, Teena Hassan, Jens U. Garbas, Michael Siebers, Ute Schmid, Michael Schöberl, Stefan Lautenbacher; BMC geriatrics 17 (2017): Problems of video-based pain detection inpatients with dementia: a road map to aninterdisciplinary solution