People on the Autism spectrum often have troubles recognizing emotions from others and knowing how to respond appropriately, which often lead to miscommunication. This project epxlores the potential solution of using facial emotion recognition to help people understand emotions of others as well as responding appropriately.


  • A pair of off-the-shelf glasses with built-in camera and speaker
  • Windows 10 IoT Core
  • Microsoft Cognitive Services Emotion API


I developed the software in C# to take images captured from the camera and used Microsoft Cognitive Services Emotions API to get emotion recognition results. I unfortunately cannot share my code as it is own by Microsoft.

Results and Analysis

After tesing the prototype with potential user, here are some of my conclusions.

Problem 1 : For users on the higher end of the Autism spectrum, a suggestive dialog support is more important because they are able to accurately identify emotions most of the time, but struggle with finding the appropriate response. Supportive dialog design is especially important for users in this group.

Problem 2 : The gesture of pressing a button for users to use this technology is a rocky integration because itfurther highlight the differences of our users from other people, which accomplishes opposite of the original goal. A possible solution is to constantly fetch for emotion reconigition results and take that feedback to let the processor decide which photo to base the suggestive dialog off of.

Problem 3 : There is also infrastructure shortcomings too. Because this project used a cloud based API, the time gap between when photos are captured and when suggestive dialog are provided is too large for this technology to be effective currently. One possible solution is to run the emotion recognition algorithm locally instead of relying on the cloud.

Problem 4 : Lastly, leaving the responsibility of capturing a photo that contains strong emotions to a targetted user that has trouble recognizing emotions is a paradox. Solution to Problem 2 could also potentially solve this problem.

As for my actual internship work that summer…