Skip to main content

Microsoft Project Oxford: Machines That Recognize Human Emotion

Microsoft Project Oxford

One of the things that separate humans from machines is that we're able to recognize and distinguish between different emotions when we interact with others. With advances in Artificial Intelligence (AI), Machine Learning and computing power in general, apps and programs are being created that are able to recognize speech, identify faces and now even distinguish between different emotions!

Microsoft Project Oxford is an evolving portfolio of APIs and SDKs that allow developers to add intelligent services to their website or build an app to provide a better experience for their users by taking advantage of Microsoft's machine learning capabilities. There are three groups of tools within the portfolio - Vision, Speech and Language.

Microsoft Project Oxford Emotion Recognition

The Emotion APIs released for public beta this week, using Microsoft's cloud-based emotion recognition algorithms, identify emotions in the faces of a photograph. The eight emotions detected are anger, contempt, disgust, fear, happiness, neutral, sadness, and surprise which are understood to be cross-culturally and universally communicated with particular facial expressions.

Once the photograph has been uploaded, the emotion recognition tool automatically identifies the faces in the photograph and then assigns a score to each of the eight emotions, with the total adding up to 1. Now that we know how the tool works, we put it to the test with some iconic images from recent history. Let's see how the tool does, shall we?

Exhibit A: President Obama Re-election Speech

President Obama Re-election Speech

 Let's start with this image from President Obama's re-election speech in November, 2012. I think it's fair to say that most of us would say he's pretty happy. Well, the Emotion Recognition Tool thinks the same way too, assigning President Obama a "happiness" score of 85%.

President Obama Re-election Speech Analyzed

OK, so that was fairly easy. What about Mitt Romney who lost the election vote to President Obama in 2012? We thought he cut a rather sad figure, but interestingly, his score for sadness was rather non-existent. Instead, he was rated by the Emotion Recognition tool as being a mix of Neutral and Happy. Maybe he was just happy the campaign was finally over! 

Mitt Romney Concession Speech Analyzed

Exhibit B: 48 Hours After The Lehman Brothers Crash

Lehman Brothers collapse Hank Paulson  and Tim Geithner

Here's an image of Hank Paulson, Ben Bernanke and Tim Geithner 48 hours after the collapse of Lehman Brothers in 2008. Ben Bernanke was almost 100% neutral but we found Tim Geithner and Hank Paulson's scores more interesting. As you can see from the above picture, Tim Geithner looked pretty angry and his scores show, with a 40% "angry" rating. Hank Paulson on the other hand had the highest "fear" rating of the 3, coming in at 10%.

Exhibit C: Afghan Girl - National Geographic Cover 

Afghan Girl National Geographic Cover

For our final image test, we decided to mix it up a little and test the iconic "Afghan Girl" image by journalist Steve McCurry, featured on the June 1985 cover of National Geographic. To us, she looked fearful and sad, with a tinge of anger in her eyes. Let's see how the Emotion Recognition Tool scores the image.

Afghan Girl National Geographic Cover Emotions Analyzed

According to the tool, her expression is mostly neutral after which "sadness" and "anger" scored the next highest, which matches our initial assessment. 


We see two main areas where the Emotion Tool can have a big impact. The first is in security - where security cameras can not only recognize faces on screen, but also identify their emotions and assess whether they pose any threat. From our experiment above, it's clear that a photograph only captures the emotions at a point in time. If the Emotion Tool could be extended to a video, this would provide more samples to draw from, allowing for a more complete assessment of the subject's emotions. 

The other area which could use the Emotion APIs is Artificial Intelligence that interacts with humans. We previously wrote about Gmail's new Smart Reply feature that could parse through the content of an email you received and based on the words and sentences, come up with a short, intelligent reply. Words form only a small part of our communication and usually more importance is placed on non-verbal cues such as our facial expressions. It would really be the next step in building smart robots if they could decode not just our words and sentence structure, but also our facial expression.

Rounding Up    

Check out the Emotion Tool and the rest of Project Oxford for some cool next-generation technology. Follows us for more interesting tech stories, cool sites and gadget buying guides!


Popular posts from this blog

How to View YouTube Shorts in the Regular YouTube Player

YouTube's "Shorts" is YouTube’s answer to TikTok and Instagram reels but let’s be honest, the YouTube Shorts player lacks some functionality, such as the ability to fast forward and go back in the video. However, you might be surprised to know that every video that loads in the "Shorts" player is actually a regular video, and by changing the URL slightly, you can load the same video in the regular player along with all the functionality of a traditional YouTube video. Changing the URL Changing the URL is the easiest way to switch the player from a technical standpoint. It’s so easy even your grandma can do it, as long as she knows how to type an address into the address bar of a browser.   Let’s use this super popular short video about AI taking over ping pong as an example. Look in the URL bar of a video playing in the "Shorts" player, where you should see the text Shorts/. Replace that text with watch?v= and load the new URL. So in this

How To Change Bluetooth Name On Android

This post will take you step-by-step through the process of changing the Bluetooth name of your devices on an Android phone. There are pictures as references in case you get lost along the way. Change Bluetooth Name On Android | Step-by-step 1. Unlock your Android phone and go to Settings Unlock your phone and click on the Settings Icon It goes without saying, but the first thing you need to do is unlock your phone. By default, the settings icon will be on your home page like in the picture above. If you don't have the settings icon, you can search for 'Settings' in the Android search bar. 2. In the Settings page, click on Bluetooth Click on the Bluetooth settings to open up the Bluetooth Settings menu Click on the Bluetooth settings which should be at the top of your Settings page. If it doesn't, scroll down till you find it or just search for it in the search bar at the top of the Settings page. 3. Click On Device Name To Change Your Phone's Bluetooth Name Click o