# Links

Youtube: <https://youtu.be/VVfNrY3ot7Y>
Vimeo: <https://vimeo.com/506690155>

# Soundtrack

Emotions and music meet to give a unique listening experience where the songs change to match your mood in real time.

## Inspiration

The last few months haven't been easy for any of us. We're isolated and getting stuck in the same routines. We wanted to build something that would add some excitement and fun back to life, and help people's mental health along the way.

Music is something that universally brings people together and lifts us up, but it's imperfect. We listen to our same favourite songs and it can be hard to find something that fits your mood. You can spend minutes just trying to find a song to listen to.

What if we could simplify the process?

## What it does

Soundtrack changes the music to match people's mood in real time. It introduces them to new songs, automates the song selection process, brings some excitement to people's lives, all in a fun and interactive way.

Music has a powerful effect on our mood. We choose new songs to help steer the user towards being calm or happy, subtly helping their mental health in a relaxed and fun way that people will want to use.

We capture video from the user's webcam, feed it into a model that can predict emotions, generate an appropriate target tag, and use that target tag with Spotify's API to find and play music that fits.

If someone is happy, we play upbeat, "dance-y" music. If they're sad, we play soft instrumental music. If they're angry, we play heavy songs. If they're neutral, we don't change anything.

## How we did it

We used Python with OpenCV and Keras libraries as well as Spotify's API.

1. Authenticate with Spotify and connect to the user's account.
2. Read webcam.
3. Analyze the webcam footage with openCV and a Keras model to recognize the current emotion.
4. If the emotion lasts long enough, send Spotify's search API an appropriate query and add it to the user's queue.
5. Play the next song (with fade out/in).
6. Repeat 2-5.

For the web app component, we used Flask and tried to use Google Cloud Platform with mixed success. The app can be run locally but we're still working out some bugs with hosting it online.

## Challenges we ran into

We tried to host it in a web app and got it running locally with Flask, but had some problems connecting it with Google Cloud Platform.

Making calls to the Spotify API pauses the video. Reducing the calls to the API helped (faster fade in and out between songs).

We tried to recognize a hand gesture to skip a song, but ran into some trouble combining that with other parts of our project, and finding decent models.

## Accomplishments that we're proud of

* Making a fun app with new tools!
* Connecting different pieces in a unique way.
* We got to try out computer vision in a practical way.

## What we learned

How to use the OpenCV and Keras libraries, and how to use Spotify's API.

## What's next for Soundtrack

* Connecting it fully as a web app so that more people can use it
* Allowing for a wider range of emotions
* User customization
* Gesture support
## Inspiration

Our idea was inspired by our group's shared interest in musical composition, as well as our interests in AI models and their capabilites. The concept that inspired our project was: "*What if life had a soundtrack?*"

## What it does

AutOST generates and produces a constant stream of original live music designed to automatically adjust to and accompany any real-life scenario.

## How we built it

We built our project in python, using the Mido library to send note signals directly to FL studio, allowing us to play constant audio without a need to export to a file. The whole program is linked up to a live video feed that uses Groq AI's computer vision api to determine the mood of an image and adjust the audio accordingly.

## Challenges we ran into

The main challenge we faced in this project is the struggle that came with making the generated music not only sound coherent and good, but also have the capability to adjust according to parameters. Turns out that generating music mathematically is more difficult than it seems.

## Accomplishments that we're proud of

We're proud of the fact that our program's music sounds somewhat decent, and also that we were able to brainstorm a concept that (to our knowlege) has not really seen much experimentation.

## What we learned

We learned that music generation is much harder than we initially thought, and that AIs aren't all that great at understanding human emotions.

## What's next for AutOST

If we continue work on this project post-hackathon, the next steps would be to expand its capabilities for recieving input, allowing it to do all sorts of amazing things such as creating a dynamic soundtrack for video games, or integrating with smart headphones to create tailored background music that would allow users to feel as though they are living inside a movie.
## Inspiration

Pusheen + Psychology + Nativescript = healthier living! Obesity is one of the leading causes of preventable death, and we wanted to help address this growing crisis.

## What it does

We wrote an iOS/Android mobile application based on psychology/neuroscience lab research dialogues that have been demonstrated to help people to improve their eating habits. The application lets people record their meals, enter mealtimes to get helpful "nudges" from Pusheen, and rate their mood before and after meals so that people can better understand the impact that their food choices are having on their life. We also include random doses of XKCD comics related to food to encourage people to browse the app for fun!

## How we built it

* "Used Heroku for Flask app deployment for backend"
* Scraped XKCD pictures and tagged captions with Monkeylearn's Machine Classifier APIs. Hand-tagged the categories that were relevant to healthy eating, and generated listing of all XKCDs suitable for healthy eating app
* Nativescript to develop mobile app
* Telerik platform to deploy app for prototyping

## Challenges we ran into

* Working with mobile apps for the first time
* Getting Nativescript dependencies and iOS/Android emulators set up on various devices
* "No success with Microsoft OCR API for reading text from xkcd comics/images (#comicsansforcaptchas)"
* Losing a laptop for several hours due to Windows 10 auto-update

## Accomplishments that we're proud of

* Working with Nativescript and Flask for the first time
* Working together with a ML API to improve image tagging quality
* Deploying our first mobile app!

## What we learned

* How to integrate different technologies to make a mobile app work

## What's next for Pusheats

* Pitching the app to friends at the psych lab!