Fraunhofer SHORE^® technology – face detection and emotion analysis that meet the highest data protection standards

Facial recognition is a hot topic right now. Already in use in numerous sectors, it is being improved all the time. But it also carries the risk of abuse, such as the systematic surveillance of millions of people. This article explains why SHORE^® is immune to such misuse, what makes the technology special and what its future looks like.

An eventful History

Sophisticated High-speed Object Recognition Engine – this is the full name of one of the most important facial recognition technologies on the market today: SHORE^®. Fraunhofer IIS began developing it back in 2005, and since then has been continuously improving SHORE^® with state-of-the-art technology. SHORE^® grew out of a predecessor project where access control was implemented based on video and audio signals. For it to work, people had to position themselves at a certain angle to the camera. But wouldn’t it be better if the face were found automatically? And so the idea of SHORE^® was born, which developed into a facial recognition technology that can not only recognize faces, but can also analyze age, gender and a large number of emotions.

The main driver of the research at the beginning was Dr. Bernhard Fröba, who laid down important groundwork in his doctoral thesis. Building on Fröba’s doctoral results, Andreas Ernst led development of the first version of the SHORE^® library, on which a highly motivated team has been working ever since. Under group manager Dr. Christian Küblbeck, pioneering work was done on developing the first real-time-capable system. His successor, Dr. Jens-Uwe Garbas, achieved considerable commercial successes through the licensing of SHORE^®. Dominik Seuß, who currently heads the Facial Analysis Solutions group, has undertaken to elevate SHORE^® to the next technological level. All these people have three things in common: a research objective that embraces human aspects; an unswerving commitment to data protection; and a highly sensitive approach to the ethical implications of facial recognition software.

Ethical Research

“Our technology has always focused on people,” says Seuß. He goes on to point out that SHORE^® was never about the narrow-minded pursuit of progress – instead, empathy and well-being were important considerations. One example is the ERIK project, where a robot was developed with real‑time feedback and emotion recognition capabilities to support the treatment of autistic children with limited social and emotional skills. In every application of SHORE^®, data protection is achieved by extracting anonymous metadata that does not allow any individual person to be identified.

Strong emphasis is placed on the context in which SHORE is used: “We seek information in advance as to how exactly the technology is to be applied,” Seuß says. It is very important to the team that the intended application be ethically and morally acceptable. Does it comply with the principles set out in the Fraunhofer Code of Conduct? Is it compatible with guidelines such as those laid down by the European Union?

“You could say that at Fraunhofer, there is always a balancing act between revenue and ethical responsibility. It’s never just about raw profit,” Seuß says. If this were the case, Fraunhofer could very well have decided for personal identification based on the recorded data. At a stroke this would have made the technology attractive to other sectors such as security and surveillance, but “crucially also much more susceptible to misuse,” Seuß points out.

A look into the future

In the future, the developers will try to incorporate even more classifiers into the SHORE^® library, but also to further develop the analysis of continuous emotional states. Take stress, for example, which is a complex human emotional state. With the aid of further information sources, however, it will be possible to analyze even this state from camera images, as Seuß reports: “A good example is the detection of heart rate, which we did in collaboration with colleagues from the Digital Health Systems department. This works based on color change per pulse wave in the face – and serves as an additional vital parameter for determining physical and emotional states.” In long-term monitoring, for example, it could provide indications of physical illnesses. Fraunhofer IIS can also rely on know‑how from other areas.[k1]  “We have colleagues in medical engineering, we have university partners – we don’t answer these questions alone,” Seuß explains. He emphasizes the good cooperation in general as well as specifically within the team. It was always an outstanding quality of the developer team, and one that led them to success: “The team members always pulled together and sought to ensure that the very best went into SHORE^®. With exemplary commitment, they poured their hearts and souls into their work.” Here’s to another successful 15 years!

For more information about SHORE^®, please visit Facial Analysis Solutions