## Inspiration

We were inspired by the challenge:
**Mirum and JWT: Can a Computer Hear How You Feel? Seeing the emotion in IM and voice.**

## What it does

By analyzing IM and voice, EmoBot will be able to score and visualize the emotional tone of a conversation.

## How we built it

**With love and a dash unicorn dust... jk,**

We built the back-end in Python. We researched API’s that analyzed text for sentiment, and used the IBM Watson Tone Analyzer API which assigns an emotion to input text. Our input text comes from a text file which we read. The Tone Analyzer assigns either anger, joy, disgust, sadness, or fear. It also tells us if the text was analytical, tentative, or confident. The API also returned a score between -1 and 1 for each emotional state to indicate magnitude. We also used a natural language API from Google to detect if a phrase is positive or negative. Google also returned a score between -1 and 1. If the score is above 0, it indicates a positive emotion. If the score is negative, that indicates a negative emotion. In python, we have our program read a text file and analyze it through both the IBM and Google APIs. We then compared the results from both Google and IBM. We used Google API as a backup to verify our results from the IBM API. If the APIs returned nothing for the outputs, we assumed that the phrases were emotionless and assigned it as “neutral”. Our program outputs the emotion associated with the text along with the score.

## Challenges we ran into

One of the challenges we faced was implementing the API into our program. We were unfamiliar with using these API and getting it to work in Python; this took some time and research. Trying to extract the correct output emotions and logistically inputting text into the API was challenging. We also needed to create a server through Flask to connect the back-end to front-end which was challenging at first because it was a new topic.

## Accomplishments that we're proud of

We made EmoBot work! And we all slept at some point during this weekend.

## What we learned

Human Emotions are complicated... very complicated.

## What's next for EmoBot

In a big picture, EmoBot can be further developed and implemented into fields like *commercial advertisement, IoT, & daily-care* etc.

More specifically, we are still in the process of improving the accuracy of the evaluation. Right now there are emotions that we are pretty confident about, but there are also a couple of feelings that are more difficult to detect. Solving issues like this can improve general user experience of EmoBot.
## Inspiration

Inspired by those with Asperger's, Sentimentex is the idea of Siyuan Liu, Faazilah Mohamed, and Saniyah Shaikh at PennApps xvi (Fall 2017). It aims to help those who have a hard time discerning the emotions from texts, emails, and other text-based media by using machine learning, natural language processing, and webscraping to calculate the percentages of emotions present in the given text.

## What it does

Sentimentex takes in text from the user and returns the probabilities of predictions for emotions present in that text, including joy, disgust, anger, fear, sadness, and guilt. Most of the current sentiment analyzers simply detect whether a sentence is positive, negative, and neutral, and we believe that a more fine-grained classifier can be useful for a variety of reasons, including for those with disabilities.

## How I built it

Over 7,000 entities of labelled data were used from a repo. A machine learning classifier, SVM, was trained on the labelled data and then used to predict the emotions for a user-inputted text. Other scores, such as the sentiment rating (whether a sentence is positive, negative, or neutral) and the sentiment score outputted by Google Natural Language Processing API, were considered as features but ultimately did not turn out to be promising. The code for the project was primarily in Python, though R, Javascript, and HTML/CSS were used as well.

## Challenges I ran into

The biggest challenge was incorporating Python with HTML using Flask. Since none of the members of the team have significant front-end experience, a fair bit of time was spent trying to get the Flask app to be hosted and work.

## Accomplishments that I'm proud of

Our model is able to predict between the 7 emotions in consideration with over 3x the accuracy of predicting at random. We are also proud of managing to integrate Python with the web application since this is the first time we have attempted to do that.

## What I learned

We learned the cases in which to use certain Machine Learning models, interacted with multiple APIs from the Google Cloud service, and learned how to use Flask and Javascript for the very first time.

## What's next for Sentimentex

All of us are very excited for the prospect of using Machine Learning and Natural Language Processing to capture the essence of written text. We plan on continuing to experiment with larger datasets and different features to improve the accuracy score of a given text, as well as to look into Speech APIs for fine-grained sentiment analysis as well.
## Inspiration

COVID revolutionized the way we communicate and work by normalizing remoteness. **It also created a massive emotional drought -** we weren't made to empathize with each other through screens. As video conferencing turns into the new normal, marketers and managers continue to struggle with remote communication's lack of nuance and engagement, and those with sensory impairments likely have it even worse.

Given our team's experience with AI/ML, we wanted to leverage data to bridge this gap. Beginning with the idea to use computer vision to help sensory impaired users detect emotion, we generalized our use-case to emotional analytics and real-time emotion identification for video conferencing in general.

## What it does

In MVP form, empath.ly is a video conferencing web application with integrated emotional analytics and real-time emotion identification. During a video call, we analyze the emotions of each user sixty times a second through their camera feed.

After the call, a dashboard is available which displays emotional data and data-derived insights such as the most common emotions throughout the call, changes in emotions over time, and notable contrasts or sudden changes in emotion

**Update as of 7.15am: We've also implemented an accessibility feature which colors the screen based on the emotions detected, to aid people with learning disabilities/emotion recognition difficulties.**

**Update as of 7.36am: We've implemented another accessibility feature which uses text-to-speech to report emotions to users with visual impairments.**

## How we built it

Our backend is powered by Tensorflow, Keras and OpenCV. We use an ML model to detect the emotions of each user, sixty times a second. Each detection is stored in an array, and these are collectively stored in an array of arrays to be displayed later on the analytics dashboard. On the frontend, we built the video conferencing app using React and Agora SDK, and imported the ML model using Tensorflow.JS.

## Challenges we ran into

Initially, we attempted to train our own facial recognition model on 10-13k datapoints from Kaggle - the maximum load our laptops could handle. However, the results weren't too accurate, and we ran into issues integrating this with the frontend later on. The model's still available on our repository, and with access to a PC, we're confident we would have been able to use it.

## Accomplishments that we're proud of

We overcame our ML roadblock and managed to produce a fully-functioning, well-designed web app with a solid business case for B2B data analytics and B2C accessibility.

And our two last minute accessibility add-ons!

## What we learned

It's okay - and, in fact, in the spirit of hacking - to innovate and leverage on pre-existing builds. After our initial ML model failed, we found and utilized a pretrained model which proved to be more accurate and effective.

Inspiring users' trust and choosing a target market receptive to AI-based analytics is also important - that's why our go-to-market will focus on tech companies that rely on remote work and are staffed by younger employees.

## What's next for empath.ly

From short-term to long-term stretch goals:

* We want to add on AssemblyAI's NLP model to deliver better insights. For example, the dashboard could highlight times when a certain sentiment in speech triggered a visual emotional reaction from the audience.
* We want to port this to mobile and enable camera feed functionality, so those with visual impairments or other difficulties recognizing emotions can rely on our live detection for in-person interactions.
* We want to apply our project to the metaverse by allowing users' avatars to emote based on emotions detected from the user.