## Overview

We created a Smart Glasses hack called SmartEQ! This unique hack leverages the power of machine learning and facial recognition to determine the emotion of the person you are talking to and conducts sentiment analysis on your conversation, all in real time! Since we couldn’t get a pair of Smart Glasses (*hint hint* MLH Hardware Lab?), we created our MVP using a microphone and webcam, acting just like the camera and mic would be on a set of Smart Glasses.

## Inspiration

Millions of children around the world have Autism Spectrum Disorder, and for these kids it can be difficult to understand the emotions people around them. This can make it very hard to make friends and get along with their peers. As a result, this negatively impacts their well-being and leads to issues including depression. We wanted to help these children understand others’ emotions and enable them to create long-lasting friendships. We learned about research studies that wanted to use technology to help kids on the autism spectrum understand the emotions of others and thought, hey, let’s see if we could use our weekend to build something that can help out!

## What it does

SmartEQ determines the mood of the person in frame, based off of their facial expressions and sentiment analysis on their speech. SmartEQ then determines the most probable emotion from the image analysis and sentiment analysis of the conversation, and provides a percentage of confidence in its answer. SmartEQ helps a child on the autism spectrum better understand the emotions of the person they are conversing with.

## How we built it

The lovely front end you are seeing on screen is build with React.js and the back end is a Python flask server. For the machine learning predictions we used a whole bunch of Microsoft Azure Cognitive Services API’s, including speech to text from the microphone, conducting sentiment analysis based off of this text, and the Face API to predict the emotion of the person in frame.

## Challenges we ran into

Newton and Max came to QHacks as a confident duo with the initial challenge of snagging some more teammates to hack with! For Lee and Zarif, this was their first hackathon and they both came solo. Together, we ended up forming a pretty awesome team :D. But that’s not to say everything went as perfectly as our new found friendships did. Newton and Lee built the front end while Max and Zarif built out the back end, and as you may have guessed, when we went to connect our code together, just about everything went wrong. We kept hitting the maximum Azure requests that our free accounts permitted, encountered very weird socket-io bugs that made our entire hack break and making sure Max didn’t drink less than 5 Red Bulls per hour.

## Accomplishments that we're proud of

We all worked with technologies that we were not familiar with, and so we were able to learn a lot while developing a functional prototype. We synthesised different forms of machine learning by integrating speech-to-text technology with sentiment analysis, which let us detect a person’s emotions just using their spoken words. We used both facial recognition and the aforementioned speech-to-sentiment-analysis to develop a holistic approach in interpreting a person’s emotions. We used Socket.IO to create real-time input and output data streams to maximize efficiency.

## What we learned

We learnt about web sockets, how to develop a web app using web sockets and how to debug web socket errors. We also learnt how to harness the power of Microsoft Azure's Machine Learning and Cognitive Science libraries. We learnt that "cat" has a positive sentiment analysis and "dog" has a neutral score, which makes no sense what so ever because dogs are definitely way cuter than cats. (Zarif strongly disagrees)

## What's next for SmartEQ

We would deploy our hack onto real Smart Glasses :D. This would allow us to deploy our tech in real life, first into small sample groups to figure out what works and what doesn't work, and after we smooth out the kinks, we could publish it as an added technology for Smart Glasses. This app would also be useful for people with social-emotional agnosia, a condition that may be caused by brain trauma that can cause them to be unable to process facial expressions.
In addition, this technology has many other cool applications! For example, we could make it into a widely used company app that company employees can integrate into their online meeting tools. This is especially valuable for HR managers, who can monitor their employees’ emotional well beings at work and be able to implement initiatives to help if their employees are not happy.
## Inspiration ✨

Seeing friends' lives being ruined through **unhealthy** attachment to video games. Struggling with regulating your emotions properly is one of the **biggest** negative effects of video games.

## What it does 🍎

YourHP is a webapp/discord bot designed to improve the mental health of gamers. By using ML and AI, when specific emotion spikes are detected, voice recordings are queued *accordingly*. When the sensor detects anger, calming reassurance is played. When happy, encouragement is given, to keep it up, etc.

The discord bot is an additional fun feature that sends messages with the same intention to improve mental health. It sends advice, motivation, and gifs when commands are sent by users.

## How we built it 🔧

Our entire web app is made using Javascript, CSS, and HTML. For our facial emotion detection, we used a javascript library built using TensorFlow API called FaceApi.js. Emotions are detected by what patterns can be found on the face such as eyebrow direction, mouth shape, and head tilt. We used their probability value to determine the emotional level and played voice lines accordingly.

The timer is a simple build that alerts users when they should take breaks from gaming and sends sound clips when the timer is up. It uses Javascript, CSS, and HTML.

## Challenges we ran into 🚧

Capturing images in JavaScript, making the discord bot, and hosting on GitHub pages, were all challenges we faced. We were constantly thinking of more ideas as we built our original project which led us to face time limitations and was not able to produce some of the more unique features to our webapp. This project was also difficult as we were fairly new to a lot of the tools we used. Before this Hackathon, we didn't know much about tensorflow, domain names, and discord bots.

## Accomplishments that we're proud of 🏆

We're proud to have finished this product to the best of our abilities. We were able to make the most out of our circumstances and adapt to our skills when obstacles were faced. Despite being sleep deprived, we still managed to persevere and almost kept up with our planned schedule.

## What we learned 🧠

We learned many priceless lessons, about new technology, teamwork, and dedication. Most importantly, we learned about failure, and redirection. Throughout the last few days, we were humbled and pushed to our limits. Many of our ideas were ambitious and unsuccessful, allowing us to be redirected into new doors and opening our minds to other possibilities that worked better.

## Future ⏭️

YourHP will continue to develop on our search for a new way to combat mental health caused by video games. Technological improvements to our systems such as speech to text can also greatly raise the efficiency of our product and towards reaching our goals!
## Inspiration

All three members of the team have had personal experiences with people on the autism spectrum, and seen firsthand their struggles with deciphering emotions. With the recent coronavirus pandemic, the use of online conference programs such as Zoom have exacerbated the issue of reading emotional cues. This is not only a problem for the 75+ million people struggling with ASD worldwide, but also for people with alexithymia, a condition that affects individuals’ ability to understand emotion (which makes up 10% of the world population). Our team decided to solve this problem through the development of Evatone, a tool that provides assistance in emotion identification on video conferencing platforms.

## What it does

Our tool is emotion detection software specifically incorporated for video conferencing. It allows users to see what major expressions they elicit as they speak and identify major emotions expressed by the other participants in the video conference. This appears as labels on video participants’ faces, allowing people with Autism and alexithymia to quickly gauge the emotions of others in the meeting room.

## How we built it

We created a live webcam feed using Typescript (part of our front end) that takes in video input. At set intervals, we send a frame from the video as a JPG image to the Python backend using Flask REST APIs. In our backend, we used the Hume Expression Measurement streaming API, along with a web socket to maintain an open connection, to analyze the facial expression of the frames in real-time, and detect the emotion. We parsed the output of Hume’s list of emotions detected to include only the emotion with the highest score (as this represented the dominant emotion), and then sent this data back to the front end (Typescript) to display it on our live webcam feed. Using cv2’s face detection model paired with div elements part of our html, we then overlay a red box over the face detected in the webcam with the dominant emotion label.

## Challenges we ran into

One challenge we ran into was ensuring that the live webcam feed was running the entire time our application was running. Originally, we built out both the live webcam feed and the backend with Python, using Python’s cv2 library to capture the frames. However, we found that the cv2 webcam frames conflicted with how we were sending frames to Hume’s API. Thus, we transitioned to using Typescript for the webcam and Python for the backend, which allowed us to run the webcam feed the whole time while simultaneously sending frames to our model.

## Accomplishments that we're proud of

We are proud of being able to connect the front end and back end of our code, seamlessly presenting our backend output and data in a more presentable way and with a much better user interface, using front-end code. We are also proud that we were able to learn how to use Hume’s API. On Saturday, we spent hours at Hume’s table debugging our code and learning how web sockets work to allow real-time continuous detection of emotion in facial expressions. We’re really proud that, while we ran into challenges with using Hume’s streaming API, we pushed through, asked questions, and got our final output!

## What we learned

We learned how to work with APIs, as we navigated the Hume API to incorporate it into our code for facial emotion detection. We learned about web sockets and how they allow for a continuous connection. We also learned how to code in TypeScript and how to use Flask as a framework for connecting HTML/CSS/JavaScript with our backend Python code using POST and GET.

## What's next for Evatone

We envision Evatone to be incorporated into online meeting platforms like Zoom, Google Meet, and Microsoft Teams. Furthermore, we see Evatone increasing accessibility for people with autism across the online sphere. We have thought about making a Chrome extension that can scan for any face currently on screen, whether that belongs to a Zoom call or a YouTube video, and similarly detect the emotion of the face, helping people with autism navigate online social interactions with ease.