## Inspiration

We were inspired to build Loki to illustrate the plausibility of social media platforms tracking user emotions to manipulate the content (and advertisements) that they view.

## What it does

Loki presents a news feed to the user much like other popular social networking apps. However, in the background, it uses iOS’ ARKit to gather the user’s facial data. This data is piped through a neural network model we trained to map facial data to emotions. We use the currently-detected emotion to modify the type of content that gets loaded into the news feed.

## How we built it

Our project consists of three parts:

1. Gather training data to infer emotions from facial expression
	* We built a native iOS application view that displays the 51 facial attributes returned by ARKit.
	* On the screen, a snapshot of the current face can be taken and manually annotated with one of four emotions [happiness, sadness, anger, and surprise]. That data is then posted to our backend server and stored in a Postgres database.
2. Train a neural network with the stored data to map the 51-dimensional facial data to one of four emotion classes. Therefore, we:
	* Format the data from the database in a preprocessing step to fit into the purely numeric neural network
	* Train the machine learning algorithm to discriminate different emotions
	* Save the final network state and transform it into a mobile-enabled format using CoreMLTools
3. Use the machine learning approach to discreetly detect the emotion of iPhone users in a Facebook-like application.
	* The iOS application utilizes the neural network to infer user emotions in real time and show post that fit the emotional state of the user
	* With this proof of concept we showed how easy applications can use the camera feature to spy on users.

## Challenges we ran into

One of the challenges we ran into was the problem of converting the raw facial data into emotions. Since there are 51 distinct data points returned by the API, it would have been difficult to manually encode notions of different emotions. However, using our machine learning pipeline, we were able to solve this.

## Accomplishments that we're proud of

We’re proud of managing to build an entire machine learning pipeline that harnesses CoreML — a feature that is new in iOS 11 — to perform on-device prediction.

## What we learned

We learned that it is remarkably easy to detect a user’s emotion with a surprising level of accuracy using very few data points, which suggests that large platforms could be doing this right now.

## What's next for Loki

Loki is currently not saving any new data that it encounters. One possibility is for the application to record the expression of the user mapped to the social media post. Another possibility is to expand on our current list of emotions (happy, sad, anger, and surprise) as well as train on more data to provide more accurate recognition. Furthermore, we can utilize the model’s data points to create additional functionalities.
## Inspiration

What if I want to take an audio tour of a national park or a University campus on my own time? What if I want to take an audio tour of a place that doesn't even offer audio tours?

With Toor, we are able to harness people's passions for the places they love to serve the curiosity of our users.

## What it does

We enable users to submit their own audio tours of the places they love, and we allow them to listen to other user submissions as well. Users can also elect to receive a text alert if a new audio tour has been updated for a specific location.

## How we built it

We built the front-end using React, and back-end with multiple REST API endpoints using Flask. Flask then uses SQLAlchemy, an ORM to submit records to the SQLite3 database and query data to and from. The audio files are stored in Google Cloud Firebase database. The front end is also hosted on Firebase.

## Challenges we ran into

Enabling users to listen to audio without having to repeatedly download the files was our first major obstacle. With some research we found that either an AWS S3 bucket or a Google Firebase database would solve our problems. After issues with permission with the AWS S3 bucket, we decided that Google Firebase would be a more apt solution to our issue.

## Accomplishments that we're proud of

Enabling audio streaming was a big win for us. We are also proud of the our team synergy and how we got things done quickly. We also are proud of the fact that we applied a lot of the things we learned from our internships this summer.

## What we learned

* Audio streaming, audio file upload
* Upload audio player on react
* Thinking about minimal viable product
* Flask
* Soft skills such as interpersonal communication with fellow hackers

## What's next for Toor

Adding the ability to comment on an audio tour, expanding the scope outside of just college campus, using Google Cloud Platform to implement Speech-To-Text and NLP to filter out "bad" comments and words in audio files.
## Inspiration

Our team sometimes gets tempered when using a computer, and we set out to minimize that. Our initial idea was related to video games, as they can often be causes of frustration, especially online ones. But we realized this idea could be scaled to cover emotions throughout all computer usage, not just video games. Whether that be working on homework, playing online games, or programming a project, there are a variety of actions you do on a computer that can get frustrating. This application seeks to alleviate.

## What it does

Once the app is open, it starts tracking your emotions via webcam, and when anger is detected, a creative calming notification will be sent to your desktop. This calming notification utilizes Google's Gemini API to generate an infinite amount of creative messages. These messages help remind the user that despite their frustration, everything is going okay and to take a break from the computer.

## How we built it

We used Python to build both the front and back end of the application. Using a model that could detect faces within a webcam, we developed a program in the backend that can predict a person's facial expression and emotion. This prediction is quite accurate with facial tracking. We then implemented Google's Gemini API as the creative source for the calming notification messages.

## Challenges we ran into

The biggest challenge we came across was finding and fine-tuning a model that catered to both our facial recognition needs and our calming message needs. Initially, we tried to use Google's MediaPipe facial detection software to track and detect faces being captured in the webcam. Unfortunately, it was incompatible with the model that we were using. Instead, we used openCV to perform facial recognition within our program. We also ran into issues with the webcam capturing weird and low-resolution angles, making it hard to capture a face within the frame. Another issue we came across was complications with Google Gemini simply not working with the API key we generated as well as text not generating correctly.

## Accomplishments that we're proud of

Getting the facial recognition working was our first hurdle that we accomplished and are extremely proud of. The fact that we were able to get our program to somewhat accurately predict facial expressions is something that we are extremely proud of. As we are both novice programmers, getting any of the above items to work was an accomplishment in and of itself.

## What we learned

We got a glimpse into the world of facial recognition and AI fine-tuning. We also learned how to utilize the Google Gemini API and integrate it into our project.

## What's next for DontClashOutAI

In the future, we hope to implement Google's Mediapipe for more advanced facial and emotion recognition. We would also like to retrain the model for more accurate and consistent results when predicting facial expressions. We're also working hard to clean up the GUI of the application allowing for a more streamlined and efficient experience.
This is where we sourced the model from: [link](https://www.kaggle.com/datasets/abhisheksingh016/machine-model-for-emotion-detection/data)