Interpreting facial expressions with AI

Most people can tell at a glance if someone looks sad, angry or happy. Researchers at Fraunhofer IIS in Erlangen used artificial intelligence to teach a software to do the same. SHORE®, the facial recognition and emotion analysis software, is now 15 years old and has been upgraded with additional functions, including heart rate recognition.

While most people can recognize and interpret facial expressions right away, children with autism frequently find this difficult. Moreover, they can often have a hard time finding the correct facial expression for what they themselves are feeling. As part of the ERIK project, therapists want to use a robot to teach autistic children how to read people’s facial expressions with a view to understanding others’ feelings.

The technology behind this work is provided by SHORE®, the facial recognition and emotion analysis software developed by Fraunhofer IIS. “SHORE® is a software library for real-time facial recognition and expression analysis using artificial intelligence methods,” says Dr. Dominik Seuß, head of the Facial Analysis Solutions group at Fraunhofer IIS. “SHORE® can recognize a person’s face and use what it sees to identify their age, sex and emotions.” The idea to incorporate facial recognition technology into therapies for autistic children came from a number of parents who wanted to make their child’s day-to-day life easier. The Fraunhofer IIS development team received a number of such inquiries. 

In collaboration with Humboldt University of Berlin and with funding from the German Federal Ministry of Education and Research (BMBF), the team got to work. The result was a white, humanoid robot with a display on its chest showing sample facial expressions that the children are invited to interpret. The robot speaks to the children, offering advice, praise, encouragement and explanations. For instance, it will tell them that a hallmark of a happy face is when the corners of the mouth are raised. Then it’s the children’s turn to imitate the expression they’ve just seen. This method appears to be effective: the participating children are finding it easier and easier to interpret other people’s facial expressions.

Keeping an eye on ethical aspects

When it comes to projects like ERIK that involve research on or with people, Fraunhofer IIS researchers always take into account the ethical, legal and social implications, or ELSI for short. They include users in the process and gain the approval of an ethics commission as a matter of course. Artificial intelligence (AI) is trained using large data sets. In the case of children’s facial expressions, the researchers train the AI using videos of people making a range of facial expressions with a view to making the training as nuanced as possible. “For every new research method, we get confirmation that what we’re doing is ethically tenable,” Seuß says. SHORE® has also held a data privacy seal for more than five years. 

Emotion recognition in advertising?

Applications for facial expression recognition are by no means limited to therapies. Take advertising: Does the key message resonate with the target group? At what point do viewers experience positive feelings, and when should the manufacturer’s or product’s logo then appear? To help answer these and similar questions, German market research company GfK also used the SHORE® facial recognition software. “GfK visited people at home and showed them a series of advertising spots on a computer and got their permission to be filmed while watching. This new footage was transmitted to the GfK offices, where it was analyzed using SHORE®,” Seuß says. 

An upgraded version of SHORE^® at 15

To mark the 15th year of SHORE®, a new version of the technology was released. It’s now possible to analyze data directly in a browser, which means that videos no longer have to leave the computer they are saved on. Instead, only the metadata, in other words the results of the analysis, are transmitted. “Now that we have a WebAssembly version, the software library can also be used in web applications. This has the enormous benefit of no longer having to transmit video material,” Seuß says.

Another enhancement is that the facial analysis can now determine heart rate. This is because each heartbeat sends a pulse wave through the body, subtly altering the color of the skin. Although these changes are imperceptible to the naked eye, the camera picks them up due to its much more finely tuned color detection. The AI is thus in a position to determine a person’s heart rate based on pulse-controlled changes to their skin tone. Potential applications here include driver monitoring systems: camera-based – and thus contactless – facial recognition can be used to capture vital parameters. Fraunhofer IIS researchers plan to develop SHORE® so that it can be used to detect even more biosignals in addition to heart rate. “To this end, we’re working closely with our neighboring Digital Health Systems department as well as with other experts whose expertise dovetails with our own,” Seuß says.

In addition, the SHORE® research team is making inroads into crown analysis. “We want to use SHORE® to detect, say, the moment when the mood of a large crowd suddenly shifts,” Seuß says. The conditions the technology supports could hardly be better: SHORE® can already detect and simultaneously analyze any number of faces featured in an image. The researchers have shown that they are open to tackling other potential applications: they have a number of ongoing projects that require them to tailor SHORE® to the needs of industry partners.