Why the Emotion AI Industry Is More Hype Than Reality

In an environment where AI may be reading facial expressions to evaluate job candidates, insurance applicants, apartment renters and others, research increasingly suggests the entire enterprise is built on false assumptions which could lead to mammoth lawsuits. JL

Lisa Barrett reports in Scientific American:

Emotion AI systems do not detect emotions. They detect physical signals, such as facial muscle movements, not the psychological meaning of those signals. It is time for emotion AI proponents and the companies that make and market these products to cut the hype and acknowledge that facial muscle movements do not map universally to specific emotions. The evidence is clear that the same emotion can accompany different facial movements and that the same facial movements can have different (or no) emotional meaning. Variety, not uniformity, is the rule.

Do your facial movements broadcast your emotions to other people? If you think the answer is yes, think again. This question is under contentious debate. Some experts maintain that people around the world make specific, recognizable faces that express certain emotions, such as smiling in happiness, scowling in anger and gasping with widened eyes in fear. They point to hundreds of studies that appear to demonstrate that smiles, frowns, and so on are universal facial expressions of emotion. They also often cite Charles Darwin’s 1872 book The Expression of the Emotions in Man and Animals to support the claim that universal expressions evolved by natural selection.

Other scientists point to a mountain of counterevidence showing that facial movements during emotions vary too widely to be universal beacons of emotional meaning. People may smile in hatred when plotting their enemy’s downfall and scowl in delight when they hear a bad pun. In Melanesian culture, a wide-eyed gasping face is a symbol of aggression, not fear. These experts say the alleged universal expressions just represent cultural stereotypes. To be clear, both sides in the debate acknowledge that facial movements vary for a given emotion; the disagreement is about whether there is enough uniformity to detect what someone is feeling.

This debate is not just academic; the outcome has serious consequences. Today you can be turned down for a job because a so-called emotion-reading system watching you on camera applied artificial intelligence to evaluate your facial movements unfavorably during an interview. In a U.S. court of law, a judge or jury may sometimes hand down a harsher sentence, even death, if they think a defendant’s face showed a lack of remorse. Children in preschools across the country are taught to recognize smiles as happiness, scowls as anger and other expressive stereotypes from books, games and posters of disembodied faces. And for children on the autism spectrum, some of whom have difficulty perceiving emotion in others, these teachings do not translate to better communication.

 

So who is right? The answer involves an unwitting physician, a scientific error and a century-long misinterpretation of Darwin’s writing. Ironically, his own observations offer a powerful resolution that is transforming the modern understanding of emotion.

The assumption of universal facial expressions can be traced back to several sources, most notably a set of photographs by 19th-century French physician Guillaume-Benjamin-Amand Duchenne. In the early days of photography, Duchenne electrically stimulated people’s facial muscles and photographed the contractions.

His photographs inspired Darwin to propose in Expression that certain facial movements were universal signs of emotion. In happiness, Darwin wrote, people smile. In sadness, they frown. The way the story is usually told, Darwin discovered that emotions have innate, biologically based expressions that are made and recognized universally and shared with other animals. That story presents facial movements as a sort of signaling system in which you can look at a person’s face, detect their emotional state and receive important information to keep you—and them—alive and healthy.

Or so it would seem. A preponderance of evidence shows that Darwin was wrong, and his mistake was a doozy. In real life, people express a given emotion with tremendous variability. In anger, for example, people in urban cultures scowl (or make some of the facial movements for a scowl) only about 35 percent of the time, according to meta-analyses of studies measuring facial movement during emotion. Scowls are also not specific to anger because people scowl for other reasons, such as when they are concentrating or when they have gas. The same tremendous variation occurs for every emotion studied—and for every other measure that purportedly tells us about someone’s emotional state, whether it’s their physiology, voice or brain activity.

Emotion AI systems, therefore, do not detect emotions. They detect physical signals, such as facial muscle movements, not the psychological meaning of those signals. The conflation of movement and meaning is deeply embedded in Western culture and in science. An example is a recent high-profile study that applied machine learning to more than six million internet videos of faces. The human raters, who trained the AI system, were asked to label facial movements in the videos, but the only labels they were given to use were emotion words, such as “angry,” rather than physical descriptions, such as “scowling.” Moreover there was no objective way to confirm what, if anything, the anonymous people in the videos were feeling in those moments.

 

There’s also considerable evidence that facial movements are just one signal of many in a much larger array of contextual information that our brain takes in. Show people a grimacing face in isolation, and they may perceive pain or frustration. But show the identical face on a runner crossing the finish line of a race, and the same grimace conveys triumph. The face is often a weaker signal of a person’s internal state than other signals in the array.

Darwin’s Expression suggests that instances of a particular emotion, such as anger, share a distinct, immutable, physical cause or state—an essence—that makes the instances similar even if they have superficial differences. Scientists have proposed a variety of essences, some of which are easily seen, such as facial movements, and others, such as complex, intertwined patterns of heart rate, breathing and body temperature, that are observed only with specialized instruments. This belief in essences, called essentialism, is compellingly intuitive. It’s also pernicious because it is virtually impossible to prove that an essence doesn’t exist. People who believe in essences but fail to observe them despite repeated attempts often continue to believe in them anyway. Researchers, in particular, tend to justify their belief by suggesting that tools and methods are not yet sufficient to locate the essences they seek.

A solution to this conundrum can be found in Darwin’s more famous book On the Origin of Species, written 13 years before Expression. Ironically, it is celebrated for helping biology “escape the paralyzing grip of essentialism,” according to heralded biologist Ernst Mayr. Before Origin was published, scholars believed that each biological species had an ideal form, created by God, with defining properties—essences—that distinguished it from all other species. Think of this as the “dog show” version of biology. In a dog show, each competitor is judged against a hypothetical ideal dog. Deviation from the ideal is considered error. Darwin’s Origin proposed, radically, that a species is a vast population of varied individuals with no essence at its core. The ideal dog doesn’t exist—it is a statistical summary of many diverse dogs. Variation is not error; it is a necessary ingredient for natural selection by the environment. When it came to emotions, however, Darwin fell prey to essentialism, ignoring his most important discovery.

 

The power of essentialism led Darwin to some beautifully ridiculous ideas about emotion, including that emotional imbalance can cause frizzy hair and that insects express fear and anger by frantically rubbing their body parts together.

Essentialism likewise appears to lure designers of emotion AI systems to follow Darwin down this comfortable path, with its assumption that emotions evolved via natural selection to serve important functions. But if you actually read Expression, you’ll find that Darwin barely mentioned natural selection. He also did not write that facial expressions are functional products of evolution. In fact, he wrote the opposite: that smiles, frowns, eye widening and other physical expressions were “purposeless”—vestigial movements that no longer serve a function. He made this statement more than 10 times in Expression. For Darwin, emotional expressions were compelling evidence that humans are animals and that we’ve evolved. By his logic, if we share expressions with other animals, but the expressions are functionally useless for us, they must have come from a long-gone, common ancestor for whom the expressions were useful.

 

Expression has been cited incorrectly for more than 100 years. How did this happen? I discovered the answer lurking in the work of an early-20th-century psychologist, Floyd Allport. In his 1924 book Social Psychology, Allport made a sweeping inference from Darwin’s writing to say that expressions begin as vestigial in newborns but quickly assume useful social functions. He wrote, “Instead of the biologically useful reaction being present in the ancestor and the expressive vestige in the descendant, we regard both these functions as present in the descendant, the former serving as a basis from which the latter develops.”

Allport’s idea, though incorrect, was attributed back to Darwin and eagerly adopted by like-minded scientists. They could now write about facial expressions as universal and claim to be the heirs of the unassailable Charles Darwin. With a single sentence, Allport misdirected the Western understanding of emotions, not only in science but in law, medicine, the eyes of the public and now emotion AI systems.

Nevertheless, this scientific tale has a happy ending because there is a name for the kind of variation we observe in real-life instances of emotion. It’s the same variation that Darwin himself observed in animal species. In Origin, Darwin described an animal species as a collection of varied individuals with no biological essence at its core. This key observation became known more generally as population thinking, and it’s supported by the modern study of genetics.

Population thinking has been revolutionizing biology for the past century, and it is now revolutionizing the science of emotion. Like a species, a given emotion such as fear, grief or elation is a vast population of varied instances. People may indeed widen their eyes and gasp in fear, but they may also scowl in fear, cry in fear, laugh in the face of fear and, in some cultures, even fall asleep in fear. There is no essence. Variation is the norm, and it is intimately linked to a person’s physiology and situation, just as variation in a species is linked to the environment its members live in.

An increasing number of emotion researchers are taking population thinking more seriously and moving beyond the essentialist ideas of the past. It is time for emotion AI proponents and the companies that make and market these products to cut the hype and acknowledge that facial muscle movements do not map universally to specific emotions. The evidence is clear that the same emotion can accompany different facial movements and that the same facial movements can have different (or no) emotional meaning. Variety, not uniformity, is the rule.

 

Darwin’s Expression is best viewed as a historical text, not a definitive scientific guide. That leads to a deeper lesson here: Science is not truth by authority. Science is the quantification of doubt by repeated observation in varied contexts. Even the most exceptional scientists can be wrong. Fortunately, mistakes are part of the scientific process. They are opportunities for discovery.