## Inspiration

We spend a lot of our time sitting in front of the computer. The idea is to use the video feed from webcam to determine the emotional state of the user, analyze and provide a feedback accordingly in the form of music, pictures and videos.

## How I built it

Using Microsoft Cognitive Services (Video + Emotion API) we get the emotional state of the user through the webcam feed. We parse that to the bot framework which in turn sends responses based upon the change in the values of emotional state.

## Challenges I ran into

Passing data between the bot framework and the desktop application which captured the webcam feed.

## Accomplishments that I'm proud of

A fully functional bot which provides feedback to the user based upon the changes in the emotion.

## What I learned

Visual studio is a pain to work with.

## What's next for ICare

Use Recurrent Neural Network to keep track of the emotional state of the user before and after and improve the content provided to the user over the period of time.
## Inspiration

What inspired us to build this application was spreading mental health awareness in relationship with the ongoing COVID-19 pandemic around the world. While it is easy to brush off signs of fatigue and emotional stress as just "being tired", often times, there is a deeper problem at the root of it. We designed this application to be as approachable and user-friendly as possible and allowed it to scale and rapidly change based on user trends.

## What it does

The project takes a scan of a face using a video stream and interprets that data by using machine learning and specially-trained models for emotion recognition. Receiving the facial data, the model is then able to process it and output the probability of a user's current emotion. After clicking the "Recommend Videos" button, the probability data is exported as an array and is processed internally, in order to determine the right query to send to the YouTube API. Once the query is sent and a response is received, the response is validated and the videos are served to the user. This process is scalable and the videos do change as newer ones get released and the YouTube algorithm serves new content. In short, this project is able to identify your emotions using face detection and suggest you videos based on how you feel.

## How we built it

The project was built as a react app leveraging face-api.js to detect the emotions and youtube-music-api for the music recommendations. The UI was designed using Material UI.

The project was built using the [REACT](https://reactjs.org/) framework, powered by [NodeJS](https://nodejs.org/en/). While it is possible to simply link the `package.json` file, the core libraries that were used were the following

* **[Redux](https://react-redux.js.org/)**
* **[Face-API](https://justadudewhohacks.github.io/face-api.js/docs/index.html)**
* **[GoogleAPIs](https://www.npmjs.com/package/googleapis)**
* **[MUI](https://mui.com/)**
* The rest were sub-dependencies that were installed automagically using [npm](https://www.npmjs.com/)

## Challenges we ran into

We faced many challenges throughout this Hackathon, including both programming and logistical ones, most of them involved dealing with React and its handling of objects and props. Here are some of the most harder challenges that we encountered with React while working on the project:

* Integration of `face-api.js`, as initially figuring out how to map the user's face and adding a canvas on top of the video stream proved to be a challenge, given how none of us really worked with that library before.
* Integration of `googleapis`' YouTube API v3, as the documentation was not very obvious and it was difficult to not only get the API key required to access the API itself, but also finding the correct URL in order to properly formulate our search query. Another challenge with this library is that it does not properly communicate its rate limiting. In this case, we did not know we could only do a maximum of 100 requests per day, and so we quickly reached our API limit and had to get a new key. Beware!
* Correctly set the camera refresh interval so that the canvas can update and be displayed to the user. Finding the correct timing and making sure that the camera would be disabled when the recommendations are displayed as well as when switching pages was a big challenge, as there was no real good documentation or solution for what we were trying to do. We ended up implementing it, but the entire process was filled with hurdles and challenges!
* Finding the right theme. It was very important to us from the very start to make it presentable and easy to use to the user. Because of that, we took a lot of time to carefully select a color palette that the users would (hopefully) be pleased by. However, this required many hours of trial-and-error, and so it took us quite some time to figure out what colors to use, all while working on completing the project we had set out to do at the start of the Hackathon.

## Accomplishments that we're proud of

While we did face many challenges and setbacks as we've outlined above, the results we something that we can really be proud of. Going into specifics, here are some of our best and satisfying moments throughout the challenge:

* Building a well-functioning app with a nice design. This was the initial goal. We did it. We're super proud of the work that we put in, the amount of hours we've spent debugging and fixing issues and it filled us with confidence knowing that we were able to plan everything out and implement everything that we wanted, given the amount of time that we had. An unforgettable experience to say the least.
* Solving the API integration issues which plagued us since the start. We knew, once we set out to develop this project, that meddling with APIs was never going to be an easy task. We were very unprepared for the amount of pain we were about to go through with the YouTube API. Part of that is mostly because of us: we chose libraries and packages that we were not very familiar with, and so, not only did we have to learn how to use them, but we also had to adapt them to our codebase to integrate them into our product. That was quite a challenge, but finally seeing it work after all the long hours we put in is absolutely worth it, and we're really glad it turned out this way.

## What we learned

To keep this section short, here are some of the things we learned throughout the Hackathon:

* How to work with new APIs
* How to debug UI issues use components to build our applications
* Understand and fully utilize React's suite of packages and libraries, as well as other styling tools such as MaterialUI (MUI)
* Rely on each other strengths
* And much, much more, but if we kept talking, the list would go on forever!

## What's next for MoodChanger

Well, given how the name **is** *Moodchanger*, there is one thing that we all wish we could change next. The world!

PS: Maybe add file support one day? :pensive:
PPS: Pst! The project is accessible on [GitHub](https://github.com/mike1572/face)!
## Inspiration

We express emotions in our everyday lives when we communicate with our loved ones, our neighbors, our friends, our local Loblaw store customer service, our doctors or therapists. These emotions can be examined by cues such as gesture, text and facial expressions. The goal of Emotional.AI is to provide a tool for businesses (customer service, etc), or doctors/therapists to identify emotions and enhance their services.

## What it does

Uses natural language processing (from audio transcription via Assembly AI) and computer vision to determine emotion of people.

## How we built it

#### **Natural Language processing**

* First we took emotion classified data from public sources (Kaggle and research studies).
* We preprocessed, cleaned, transformed, created features, and performed light EDA on the dataset.
* Used TF-IDF tokenizer to deal with numbers, punctuation marks, non letter symbols, etc.
* Scaled the data using Robust Scaler and made 7 models. (MNB, Linear Regression, KNN, SVM, Decision Tree, Random Forrest, XGB)

#### **Computer Vision**

Used Mediapipe to generate points on face, then use those points to get training data set. We used Jupyter Notebook to run OpenCV and Mediapipe. Upon running our data in Mediapipe, we were able to get a skeleton map of the face with 468 points. These points can be mapped in 3-dimension as it contains X, Y, and Z axis. We processed these features (468 points x 3) by saving them into a spreadsheet. Then we divided the spreadsheet into training and testing data. Using the training set, we were able to create 6 Machine learning models and choose the best one.

#### **Assembly AI**

We converted video/audio from recordings (whether it’s a therapy session or customer service audio from 1000s of Loblaws customers 😉) to text using Assembly API.

#### **Amazon Web Services**

We used the S3 services to host the video files uploaded by the user. These video files were then sent the Assembly AI Api.

#### **DCP**

For Computing (ML)

## Challenges we ran into

* Collaborating virtually is challenging
* Deep learning training takes a lot of computing power and time
* Connecting our front-end with back-end (and ML)
* Time management
* Working with react + flask server
* Configuring amazon buckets and users to make the app work with the s3 services

## Accomplishments that we're proud of

Apart from completing this hack, we persevered through each challenge as a team and succeeded in what we put ourselves up to.

## What we learned

* Working as a team
* Configuration management
* Working with Flask

## What's next for Emotional.AI

* We hope to have a more refined application with cleaner UI.
* We want to train our models further with more data and have more classifications.
* We want to make a platform for therapists to connect with their clients and use our tech.
* Make our solution work in real-time.