How Facebook, A Pixar Artist, And Charles Darwin Are Reinventing The Emoticon

What comes after the smiley?

Facebook wants an awful lot from its emoticons: They should be able to convey complex emotions, for example, like contemplation, admiration, affirmation, maternal love, determination, devotion, resignation, and gratitude.

But how, in a tiny digital image, do you depict something as subtle as shame as opposed to remorse, or shyness as opposed to modesty? Current emoticons can’t do that, or anything close to it. So Facebook has turned to Pixar story illustrator and former storyboard artist at the Wallace and Gromit studio Matt Jones, to help make something entirely new. He’s charged, basically, with reinventing the smiley.

“Facebook was canny enough to realize that traditional emoticons are quite bland,” says Jones. “At Pixar we consider emotional states every day with every drawing we make. Our work is informed by the years of study we do, constantly studying people’s gestures and expressions in real life.” (To be clear, it isn’t an official collaboration between Pixar and Facebook. Jones is working independently.)

Jones was studying facial expressions for his work on a new film by Pete Docter (director of Up and Monsters, Inc.), which takes place in the mind of a young girl coming of age. The film, referred to as The Untitled Pixar Movie That Takes You Inside the Mind, is set for release in June 2015. Pixar is famous for bringing to life inanimate, and even mundane, objects: lamps, cars, a hockey puck. This film required an extra level of emotional detail. A young girl’s mental state is a complicated thing.

Docter brought in psychologist Paul Ekman, a pioneer in the research of facial expressions, as a consultant on micro-emotions, the small mini expressions that happen between more major ones. (You may have heard of Ekman as John Cleese’s collaborator on the BBC series, The Human Face.)

At the same time, Ekman’s protégé, Dacher Keltner, codirector of University of California-Berkeley’s Greater Good Science Program, was starting work with Facebook to improve their emoticons. When Keltner heard about the project at Pixar, he approached the company. That’s how he found Jones.

Keltner started off by giving Jones some of the classic universal emotions (anger, disgust, fear, happiness, sadness, and surprise) to translate into emoticon-style drawings. He loved what he saw — and decided to up the ante. He handed over Charles Darwin’s The Expression of the Emotions in Man and Animals, which Darwin published 13 years after On the Origin of Species. The book explores similarities between human and animal facial movements, in support of Darwin’s theory that humans and animals have a common ancestor. It became one of the seminal works on the facial emotive expressions.

Photographs of emotional expressions Darwin used in his research:

Via: en.wikipedia.org

Some of Darwin’s emotions come with elaborate definitions, but some get descriptions of just two or three words. “It was a good graphic-design cartoon challenge to draw what he was thinking,” says Jones. Some of Darwin’s findings were too antiquated. For instance, he described the expression of embarrassment as a little cough. “It’s very Victorian. If we are embarrassed, these days we are bit broader,” Jones says, “especially Americans.”

Not all of Darwin’s facial expressions are scientifically supported. Research is still being done to show whether the feelings Darwin outlined translate into universal facial movements — Keltner, for his part, added some extra emotions to the list. This leaves Jones, as an artist, exploring the same correlations between feelings and muscle movement as leading psychologists.

“I was skeptical when I handed Matt these emotions, like gratitude, relief, awe, and guilt, which no one has studied,” says Keltner. But Jones delivered: “He does simple lines for facial muscle movements, and when you look at them, you just say, ‘wow,’ because there is so much expressiveness.”

“If we can crack a universal language, that will be true success,” says Jones. “What we need to aim at is instant readability, just like what we do in cartoons.”

Jones’ favorites are the faces expressing negative emotions, like disgust, since it’s a good chance to “draw a gross cartoon character.” By contrast, the positive ones, like maternal love, are subtler and hence more difficult.

You can see the influence of his experience working on films such as Pirates! Band of Misfits at the Wallace and Gromit studios, in the big, googly eyes, wide expressive mouths with teeth, and exaggerated eyebrows. “As long as you have eyebrows, you are safe,” he says, a lesson he learned from Gromit, whose rubbery brow was a trademark.

He tried adding noses (for sneering) and shoulders (for shrugging), but they don’t quite work on the emoticon scale. Instead of a shrug of the shoulder, he’ll use a tilt of the head, a curled bottom lip, or a raised eyebrow.

Keltner has been testing Jones’ drawings out on subjects around the world. In a sort of back-to-front approach, he is also translating Jones’ simple line drawings back into anatomical coding of real facial movements to see if they work on a human face.

Right now, Jones is experimenting with colors beyond the default yellow used in most other emoticons. He tried “Facebook blue,” thinking it might have become familiar enough to users, but said the emoticons just looked like they had hypothermia. He’s experimenting with multiple colors: red for anger, green for envy. “But you don’t want to offend anyone,” he explains. “Colors will be a racial issue.”

Jones is considering whether emoticons can be three-dimensionally turned to the side, to indicate things like love or devotion. “Holding your head straight forward is an emotional reflex in itself,” he points out. “It’s quite rigid and I’m trying to break that.” He is considering whether the face has to live within the confines of the circle, or if eyebrows can float above, or tears fly out.

And of course, as a Pixar person, he is pushing for animation. “I love on Facebook when you hit the ‘like’ button on your iPhone, and the thumb will kind of pulse a little,” says Jones. “I’m encouraging them to put movement into the emoticons. Hopefully we can evolve them into living little living characters.”

The biggest challenge they are facing is how to simplify and shrink down the illustrations to chat-box size — ultimately, these will be just a few pixels wide. “We are trying to figure out how much looseness we can keep,” explains Jones, “so they don’t lose the expressions.”

Correction: Jones worked on the movie Pirates! Band of Misfits.