## Check it out on GitHub!

The machine learning and web app segments are split into 2 different branches. Make sure to switch to these branches to see the source code! You can view the repository [here](https://github.com/SuddenlyBananas/be-right-back/).

## Inspiration

Inspired in part by the Black Mirror episode of the same title (though we had similar thoughts before we made the connection).

## What it does

The goal of the project is to be able to talk to a neural net simulation of your Facebook friends you've had conversations with. It uses a standard base model and customizes it based on message upload input. However, we ran into some struggles that prevented the full achievement of this goal.

The user downloads their message history data and uploads it to the site. Then, they can theoretically ask the bot to emulate one of their friends and the bot customizes the neural net model to fit the friend in question.

## How we built it

Tensor Flow for the machine learning aspect, Node JS and HTML5 for the data-managing website, Python for data scraping. Users can interact with the data through a Facebook Messenger Chat Bot.

## Challenges we ran into

AWS wouldn't let us rent a GPU-based E2 instance, and Azure didn't show anything for us either. Thus, training took much longer than expected.

In fact, we had to run back to an apartment at 5 AM to try to run it on a desktop with a GPU... which didn't end up working (as we found out when we got back half an hour after starting the training set).

The Facebook API proved to be more complex than expected, especially negotiating the 2 different user IDs assigned to Facebook and Messenger user accounts.

## Accomplishments that we're proud of

Getting a mostly functional machine learning model that can be interacted with live via a Facebook Messenger Chat Bot.

## What we learned

Communication between many different components of the app; specifically the machine learning server, data parsing script, web server, and Facebook app.

## What's next for Be Right Back

We would like to fully realize the goals of this project by training the model on a bigger data set and allowing more customization to specific users.
## Inspiration

COVID revolutionized the way we communicate and work by normalizing remoteness. **It also created a massive emotional drought -** we weren't made to empathize with each other through screens. As video conferencing turns into the new normal, marketers and managers continue to struggle with remote communication's lack of nuance and engagement, and those with sensory impairments likely have it even worse.

Given our team's experience with AI/ML, we wanted to leverage data to bridge this gap. Beginning with the idea to use computer vision to help sensory impaired users detect emotion, we generalized our use-case to emotional analytics and real-time emotion identification for video conferencing in general.

## What it does

In MVP form, empath.ly is a video conferencing web application with integrated emotional analytics and real-time emotion identification. During a video call, we analyze the emotions of each user sixty times a second through their camera feed.

After the call, a dashboard is available which displays emotional data and data-derived insights such as the most common emotions throughout the call, changes in emotions over time, and notable contrasts or sudden changes in emotion

**Update as of 7.15am: We've also implemented an accessibility feature which colors the screen based on the emotions detected, to aid people with learning disabilities/emotion recognition difficulties.**

**Update as of 7.36am: We've implemented another accessibility feature which uses text-to-speech to report emotions to users with visual impairments.**

## How we built it

Our backend is powered by Tensorflow, Keras and OpenCV. We use an ML model to detect the emotions of each user, sixty times a second. Each detection is stored in an array, and these are collectively stored in an array of arrays to be displayed later on the analytics dashboard. On the frontend, we built the video conferencing app using React and Agora SDK, and imported the ML model using Tensorflow.JS.

## Challenges we ran into

Initially, we attempted to train our own facial recognition model on 10-13k datapoints from Kaggle - the maximum load our laptops could handle. However, the results weren't too accurate, and we ran into issues integrating this with the frontend later on. The model's still available on our repository, and with access to a PC, we're confident we would have been able to use it.

## Accomplishments that we're proud of

We overcame our ML roadblock and managed to produce a fully-functioning, well-designed web app with a solid business case for B2B data analytics and B2C accessibility.

And our two last minute accessibility add-ons!

## What we learned

It's okay - and, in fact, in the spirit of hacking - to innovate and leverage on pre-existing builds. After our initial ML model failed, we found and utilized a pretrained model which proved to be more accurate and effective.

Inspiring users' trust and choosing a target market receptive to AI-based analytics is also important - that's why our go-to-market will focus on tech companies that rely on remote work and are staffed by younger employees.

## What's next for empath.ly

From short-term to long-term stretch goals:

* We want to add on AssemblyAI's NLP model to deliver better insights. For example, the dashboard could highlight times when a certain sentiment in speech triggered a visual emotional reaction from the audience.
* We want to port this to mobile and enable camera feed functionality, so those with visual impairments or other difficulties recognizing emotions can rely on our live detection for in-person interactions.
* We want to apply our project to the metaverse by allowing users' avatars to emote based on emotions detected from the user.
## Inspiration

Many visually impaired people have difficulties on a day-to-day basis reading text and other items, and other comparable technologies and software are either expensive or bulky and impractical to use. We sought to create a cheap, light and reliable system for text reading.

## What it does

A wearable that reads aloud text to its user through a mobile app. It was made for the visually impaired who have trouble reading text that doesn't have a braille translation.

## How we built it

We used an ESP-32 Cam, mounted in a 3d printed enclosure atop a hat. The camera hosts a WiFi Local Access Point, to which the user's smartphone is connected to. The smartphone processes the image with an OCR, and sends the resulting text to a text-to-speech API, which is played to the user.

## Challenges we ran into

* We forgot to bring a charger and HDMI cable for our raspberry PI and were unable to rely on its processing power.
* We tried to run ML models on the ESP32-CAM but we were only able to small and simple models due to the hardware limitations.
* We were unable to send images over Bluetooth LE because of the low data transfer limit and we were unable to connect the ESP32-CAM so we opted to create a soft access point on the esp32 so other devices could connect to it to retrieve images.
* Getting a 3D profile of the hat (we borrowed **stole** from xero) was challenging because we couldn't measure the spherical shape of the hat.
* We had a lot of trouble retrieving images on our React Native app, making us switch to React. After a lot of trial and error, we finally got the ESP32-CAM to stream images on a ReactJS website, but we didn't have enough time to combine the text recognition with it.

## Accomplishments that we're proud of

* 3d Printed Shroud
* Building a website and implemented Machine Learning model
* Streaming ESP32-CAM to our website

## What we learned

* Always allow more overhead when possible to allow for changes while creating your project
* Create a schedule to predict what you will have time for and when it should be done by
	+ ML training is very time-consuming and it is very likely that you will have bugs to work out
	+ Start with a smaller project and work up towards a larger and more complete one
* New things our team members have experienced: React (JS & Native), ESP32-CAM, Tesseract (js & py), NodeJS, 3d modeling/printing under a time constraint, getting free food

## What's next for Assisted Reader

* Fix issues / merge the ESP32-CAM with the ML models
* Better form factor (smaller battery, case, lower power usage)
* Use automatic spelling correction to ensure Text-to-Speech always reads proper English words
* More ML training for an improved OCR model
* Translation to other languages for a larger customer base
* Cleaner and more modern User Interface
* Add support for Bluetooth and connecting to other Wifi networks