Microsoft Corporation, having launched a service that allows you to determine, decided to go further. Now the company's specialists have developed an algorithm that determines the emotions of the person depicted in the photo. In total, the test is carried out in eight positions: anger, satisfaction, disgust, fear, happiness, neutrality (we read - a detached facial expression), sadness, surprise.
The developers claim that the results are not very accurate, but it is clear that the assessment is more accurate than in the case of age determination. As an example, many of us can take the famous photograph of Keanu Reeves sitting on a bench and chewing something. Microsoft believes that such an emotion as sadness is almost absent on the actor’s face (0.01831), his facial expression is neutral.
Another famous photograph shows us a child whose emotional state is rated as “angry.” Here we have as much as 0.98752 points.
The company's work on such algorithms is not a game. The corporation is trying to improve the software responsible for recognizing photos. Despite some errors (especially with age determination), everything looks quite interesting. Microsoft plans to open an API for its image services, so it will be possible to create software and web services that sort photos based on the emotions of the people captured in the pictures.
Let's see what emotions Bill Gates feels 
Tim Cook 
Heroes of Fallout 4 
Do you want to try the resource in action? No problem, here you go
Instructions. I have a very interesting game for you. You know that all people are sometimes happy, sometimes surprised or angry. Now I will show you some photographs that depict a person with some kind of emotion on his face (for example, joy, grief, fear, surprise, contempt, disgust, shame, interest, anger). Here's a list of these emotions. Try to determine what emotion is reflected in each photo.
The experimenter shows 9 photographs one by one and writes down the child’s answer on the form next to the photo number.
Form for registering answers
| Order number | Photo | Subject's answer | Note |
622 Application
The number of correct answers is taken into account, which is used to judge the developed ability to determine the sign of an emotion by facial expression.
Determining emotions by pantomime
Instructions. And now I will show you pictures that also depict a person with some kind of experiences. Try to determine what emotion the person has in each picture.
The subject is shown 8 pictures in turn and writes down his answers on the form.
Form for registering answers
| Order number | Picture | Subject's answer | Notes |
| M+. +.IP--MM | |||
The number of correct answers is taken into account and the developed ability to determine the sign of emotion by pantomime is judged from them.
Studying the perception of the expressive side of the drawing
The technique is taken from the article: Repina T. A. Preschoolers’ perception of the expressive side of a drawing and its influence on children’s attitude towards the hero of the book // Questions of psychology. - 1960. - No. 5.
Preparation of the study. Select 10-14 reproductions of paintings in which the emotional content is conveyed by showing facial expressions and gestures, the poses of the characters, the relationships of the characters with each other or their relationship to surrounding objects.
Conducting research. The experiment is carried out individually with children 3-6 years old. The child is shown pictures sequentially and asked to talk about them. If he finds it difficult to describe, they ask leading questions (prepare in advance) to find out how he understands the plot of the image and the emotional states of the characters.
Data processing. The number of correct answers is counted when the children correctly understood the plot and the emotional state of the characters. Counting is carried out for each picture. The results are presented in a table by age: 3-4 years, 4-5 years, 5-6 years, 6-7 years.
Find out which emotional states depicted in the pictures are correctly perceived by children and at what age. How do they perceive facial expressions of joy and anger, expressions of sadness and sadness, with what emotional experience connect the bright colors of the drawing, as they understand the emotional content conveyed through the pose and gestures of the hero.
Studying the emotional manifestations of children when acting out story scenes
The technique is taken from the book: Emotional development of a preschooler / Ed. A. D. Koshelevoy. - M., 1985. - P. 100-102.
Preparation of the study. Select situations from the lives of children that are close and understandable to them, for example:
Appendix 623
1. Sick mother lies in bed, eldest daughter ( preparatory group) brings his brother (nursery group).
2. During lunch in a group, a boy accidentally spills soup, all the children jump up and laugh; the boy is scared, the teacher sternly explains that he needs to be careful and that there is absolutely nothing to laugh about here.
3. The boy lost his mittens, and his hands became very cold during a walk, but he does not want to show others that he is very cold.
4. The girl was not accepted into the game, she went to the corner of the room, hung her head low and was silent, about to cry.
5. A boy (girl) is happy for his friend (girlfriend), whose drawing turned out to be the best in the group.
Prepare 2-3 children to dramatize these stories. Conducting research. The study is conducted with children 4-7 years old. First episode. Children prepared in advance act out a skit in front of the group, then the experimenter asks the children how the characters in this skit feel. Second series. The experimenter describes the situation and offers to depict it.
I situation - show the sad, suffering face of the mother, the capricious crying boy and the sympathetic face of the girl.
II situation - show the stern face of the teacher, the laughing and then embarrassed children, the frightened face of the boy.
III situation - how the boy does not want to show that he is cold.
IV situation - show the girl's resentment.
Vsituation - show genuine joy for another. If children do not expressively or incorrectly portray the feelings and emotions of the characters, the experimenter again describes the situations and tells in detail what each of the characters is experiencing.
Data processing. They analyze how children embody the emotional states of characters in sketches. They draw a conclusion about the expressiveness and richness of expressive and facial means of communication and about the development of the ability to empathize with other people. The data is presented in a table (Table 6).
Today, more and more aspects of our lives are subject to automation and mathematical processing, one of which is the main part of our memory - photographs that depict us, our friends, memorable moments and much more. Not so long ago, Microsoft already took the first and rather interesting step in photo processing - an algorithm was developed that made it possible to determine the age of the captured people (). But time does not stand still, and today we will talk about a new algorithm proposed by Microsoft - recognizing emotions from photos.
Subconsciously, we all know how to determine whether a person in a photo is laughing or sad, but is it possible to measure these feelings? It turns out it is possible! And let's immediately see what this measurement is.
First, go to the project page https://www.projectoxford.ai/demo/emotion#detection.
And we immediately see an example of how the algorithm works. Let's try to figure out what it is, and then we'll conduct some experiments.
So, on the left there is an analyzed image in which faces are selected. It immediately becomes clear that the algorithm can identify all faces in photographs and analyze each one.
On the right (and when you hover over the frames on faces) there is an area containing a bunch of incomprehensible text (the specified text is the result of the algorithm in a special format), let's break it down.
In the first line there is the inscription “detection result” - “analysis results”.
"FaceRectangle": (
"Left": 488,
"Top": 263,
"Width": 148,
"Height": 148
"Scores": (
"Anger": 9.075572e-13,
"Contempt": 7.048959e-9,
"Disgust": 1.02152783e-11,
"Fear": 1.778957e-14,
"Happiness": 0.9999999,
"Neutral": 1.31694478e-7,
"Sadness": 6.04054263e-12,
"Surprise": 3.92249462e-11
What does all this mean?
In the “FaceRectangle” fragment, the coordinates of the corners of the frame of each of the faces are indicated, so that it is clear which person’s emotions we will look at.
The “Scores” fragment presents the results themselves. Because The algorithm is currently provided in trial mode; the results have not been processed into a “human” form and are presented in “raw form”. But don’t be scared - everything is very simple! There is a list of eight emotions:
Anger - Anger
Contempt - Contempt
Disgust - Disgust
Fear- Fear
Happiness - Happiness
Neutral - Neutral
Sadness - Sadness
Surprise - surprise
And the contribution of each emotion to the general emotional state of a person in the photo is indicated from 0 to 1 - the higher the number, the more significant the emotion is.
In the example considered, we see that the first person in the photo has the most pronounced emotion - happiness. Don’t be alarmed by incomprehensible values like “3.92249462e-11” - this is a mathematical form of writing a number, where “e-11” indicates how many places to the left the decimal point should be moved, i.e. in the example the number “0.0000000000392249462” will be obtained, which is incredibly small.
Now you can fantasize about using an emotion recognition algorithm from photos. For example, I see a photo album on a social network that, depending on the user’s chosen status (sadness, joy or boredom), automatically rearranges photos to suit the mood. Or another example, you are going to look at pictures on the Internet, the camera of your device takes a picture of you and, according to the specified algorithm, your mood is determined, and then images that correspond to your emotional state are offered for viewing, for example, if you are sad, the images will be colorful and positive.
I'm sure the algorithm will find many uses. Moreover, Microsoft has provided an API for accessing the algorithm, which means that anyone can use it and provide processing results in a variety of ways.
Well, finally, I dabbled a little with automatic recognition of emotions and want to show what came out of it.
The first photo I checked was, of course, the famous Terminator - Mr. Deadpan. And the algorithm supported me - 0.89 neutrality.
Why not try something fun? And who is the best at making faces and having fun is Musk’s first thought. And unanimously - a face full of “happiness”.
Since the movies are on, we should try the Hobbit. Here I myself cannot determine the whole range of emotions on the face and the algorithm agrees with me. However, at that moment I didn’t notice an interesting thing - if you hover the cursor over the frame in the photo, a window with the results appears, a nice little thing, especially when there are a lot of faces.
And again Jim Carrey, well, it seems to me that he has very vivid emotions - and we look at surprise. It seems I was right.
In the photo there is a young man whose emotional state we defined as angry.
Let’s complicate the task and give the algorithm pre-prepared photos of emotions. Faces, of course, are not entirely natural for clearly conveying the state, but everything was determined with very high accuracy.
Well, let's cheat a little. Not a photo, but a portrait drawn by an artist from our website () and I think this is also a good result!
Of course, I couldn’t resist and tried to recognize the feelings of Cheburashka and was left with nothing - the algorithm refused to process the cartoon character.
Summing up the review of the emotion recognition algorithm from photos, implemented by Microsoft, I would like to note very good results his work and a large number of accurate recognitions. Just a great result for a demo start! We will wait for new interesting solutions.
Instructions. I have a very interesting game for you. You know that all people are sometimes happy, sometimes surprised or angry. Now I will show you some photographs that depict a person with some kind of emotion on his face (for example, joy, grief, fear, surprise, contempt, disgust, shame, interest, anger). Here's a list of these emotions. Try to determine what emotion is reflected in each photo.
The experimenter shows 9 photographs one by one and writes down the child’s answer on the form next to the photo number.
Form for registering answers
| Order number | Photo | Subject's answer | Note |
622 Application
The number of correct answers is taken into account, which is used to judge the developed ability to determine the sign of an emotion by facial expression.
Determining emotions by pantomime
Instructions. And now I will show you pictures that also depict a person with some kind of experiences. Try to determine what emotion the person has in each picture.
The subject is shown 8 pictures in turn and writes down his answers on the form.
Form for registering answers
| Order number | Picture | Subject's answer | Notes |
| M+. +.IP--MM | |||
The number of correct answers is taken into account and the developed ability to determine the sign of emotion by pantomime is judged from them.
Studying the perception of the expressive side of the drawing
The technique is taken from the article: Repina T. A. Preschoolers’ perception of the expressive side of a drawing and its influence on children’s attitude towards the hero of the book // Questions of psychology. - 1960. - No. 5.
Preparation of the study. Select 10-14 reproductions of paintings in which the emotional content is conveyed by showing facial expressions and gestures, the poses of the characters, the relationships of the characters with each other or their relationship to surrounding objects.
Conducting research. The experiment is carried out individually with children 3-6 years old. The child is shown pictures sequentially and asked to talk about them. If he finds it difficult to describe, they ask leading questions (prepare in advance) to find out how he understands the plot of the image and the emotional states of the characters.
Data processing. The number of correct answers is counted when the children correctly understood the plot and the emotional state of the characters. Counting is carried out for each picture. The results are presented in a table by age: 3-4 years, 4-5 years, 5-6 years, 6-7 years.
Find out which emotional states depicted in the pictures are correctly perceived by children and at what age. How they perceive facial expressions of joy and anger, expressions of sadness and sadness, what emotional experience they associate with the bright colors of the drawing, how they understand the emotional content conveyed through the hero’s posture and gestures.
Studying the emotional manifestations of children when acting out story scenes
The technique is taken from the book: Emotional development of a preschooler / Ed. A. D. Koshelevoy. - M., 1985. - P. 100-102.
Preparation of the study. Select situations from the lives of children that are close and understandable to them, for example:
Appendix 623
1. A sick mother lies in bed, the eldest daughter (preparatory group) brings her brother (nursery group).
2. During lunch in a group, a boy accidentally spills soup, all the children jump up and laugh; the boy is scared, the teacher sternly explains that he needs to be careful and that there is absolutely nothing to laugh about here.
3. The boy lost his mittens, and his hands became very cold during a walk, but he does not want to show others that he is very cold.
4. The girl was not accepted into the game, she went to the corner of the room, hung her head low and was silent, about to cry.
5. A boy (girl) is happy for his friend (girlfriend), whose drawing turned out to be the best in the group.
Prepare 2-3 children to dramatize these stories. Conducting research. The study is conducted with children 4-7 years old. First episode. Children prepared in advance act out a skit in front of the group, then the experimenter asks the children how the characters in this skit feel. Second series. The experimenter describes the situation and offers to depict it.
I situation - show the sad, suffering face of the mother, the capricious crying boy and the sympathetic face of the girl.
II situation - show the stern face of the teacher, the laughing and then embarrassed children, the frightened face of the boy.
III situation - how the boy does not want to show that he is cold.
IV situation - show the girl's resentment.
Vsituation - show genuine joy for another. If children do not expressively or incorrectly portray the feelings and emotions of the characters, the experimenter again describes the situations and tells in detail what each of the characters is experiencing.
Data processing. They analyze how children embody the emotional states of characters in sketches. They draw a conclusion about the expressiveness and richness of expressive and facial means of communication and about the development of the ability to empathize with other people. The data is presented in a table (Table 6).
Stimulus material: a set of colored cards from the Luscher test is used, designated by numbers: 0 - gray, 1 - blue, 2 - green, 3 - red, 4 - yellow, 5 - purple, 6 - brown, 7 - black.
The following list is used:
1) my mother,
2) my dad,
3) brother or sister (if the child has one),
4) my mood at school,
5) my mood at home,
7) teacher,
Order of conduct reminds me of a game. All 8 cards are laid out in random order in front of the child on a white background.
Instructions: “Please tell me which color is most suitable to represent your mother?” etc. according to the list. Next, the colors are ranked by the child in order of preference, starting with the most “beautiful, pleasant” and ending with the most “ugly, unpleasant.” Thus, each person on the list receives his own rank designation.
The higher the level of emotional attractiveness, intimacy, sympathy and less conflict in relationships with someone, the more preferred color these people should be associated with by the child. On the contrary, a negative, hostile attitude towards someone is indicated by the color assigned to him, which received the most negative rank in the color layout.
APPENDIX 2
Methodology “Studying the understanding of the emotional states of people depicted in the picture”
Stimulus material: portrait pictures (photos) depicting children and adults who have a clearly expressed emotional state of both basic emotions (joy, fear, anger, grief) and their shades; story pictures depicting positive and negative actions of children and adults.
Order of conduct. Two series are conducted individually with children of primary school age.
First episode. Instructions: the child is shown sequentially portrait pictures of children and adults and asked the questions: “Who is shown in the picture? What is he doing? How is he feeling? How did you figure this out? Describe what you see in the picture.”
Second series. Instructions: the child is shown sequential pictures and asked questions: “What are the children (adults) doing? How do they do it (friendly, quarrel, do not pay attention to each other, etc.)? How did you guess? Which one is good and which one is bad? How did you guess?"
Processing of results and interpretation. Count the number of correct answers in different age groups separately for each series and for each picture. Determine whether children understand emotional states adults and peers, by what signs they are guided, who they understand better - an adult or a peer. The dependence of these indicators on the age of the children is determined.
APPENDIX 3
Methodology “Determination of emotions from photographs”
Stimulus material: a set of 9 photographs depicting a person whose face reflects one or another emotion, and a list of emotions.
Order of conduct. Instructions: “I have a very interesting task. You know that all people are sometimes happy, sometimes surprised or angry. Now I will show you several photographs that depict a person in a certain emotional state (for example, joy, grief, fear, surprise, contempt, disgust, shame, interest, anger). Here's a list of these emotions. Try to determine exactly what emotion is reflected in each photograph.”
The experimenter shows 9 photographs one by one and writes down the child’s answer on the form next to the photo number.
Form for registering answers
|
Subject's answer |
Note |
||
Processing of results and interpretation. The number of correct answers is taken into account, which is used to judge the maturity of the ability to determine a person’s emotion by facial expression. 50% or less correct answers indicate a very low level of ability to distinguish between emotions.