## Inspiration

2 days before flying to Hack the North, Darryl forgot his keys and spent the better part of an afternoon retracing his steps to find it- But what if there was a personal assistant that remembered everything for you? Memories should be made easier with the technologies we have today.

## What it does

A camera records you as you go about your day to day life, storing "comic book strip" panels containing images and context of what you're doing as you go about your life. When you want to remember something you can ask out loud, and it'll use Open AI's API to search through its "memories" to bring up the location, time, and your action when you lost it. This can help with knowing where you placed your keys, if you locked your door/garage, and other day to day life.

## How we built it

The React-based UI interface records using your webcam, screenshotting every second, and stopping at the 9 second mark before creating a 3x3 comic image. This was done because having static images would not give enough context for certain scenarios, and we wanted to reduce the rate of API requests per image. After generating this image, it sends this to OpenAI's turbo vision model, which then gives contextualized info about the image. This info is then posted sent to our Express.JS service hosted on Vercel, which in turn parses this data and sends it to Cloud Firestore (stored in a Firebase database). To re-access this data, the browser's built in speech recognition is utilized by us along with the SpeechSynthesis API in order to communicate back and forth with the user. The user speaks, the dialogue is converted into text and processed by Open AI, which then classifies it as either a search for an action, or an object find. It then searches through the database and speaks out loud, giving information with a naturalized response.

## Challenges we ran into

We originally planned on using a VR headset, webcam, NEST camera, or anything external with a camera, which we could attach to our bodies somehow. Unfortunately the hardware lottery didn't go our way; to combat this, we decided to make use of MacOS's continuity feature, using our iPhone camera connected to our macbook as our primary input.

## Accomplishments that we're proud of

As a two person team, we're proud of how well we were able to work together and silo our tasks so they didn't interfere with each other. Also, this was Michelle's first time working with Express.JS and Firebase, so we're proud of how fast we were able to learn!

## What we learned

We learned about OpenAI's turbo vision API capabilities, how to work together as a team, how to sleep effectively on a couch and with very little sleep.

## What's next for ReCall: Memories done for you!

We originally had a vision for people with amnesia and memory loss problems, where there would be a catalogue for the people that they've met in the past to help them as they recover. We didn't have too much context on these health problems however, and limited scope, so in the future we would like to implement a face recognition feature to help people remember their friends and family.
## Inspiration

We wanted to create a webapp that will help people learn American Sign Language.

## What it does

SignLingo starts by giving the user a phrase to sign. Using the user's webcam, gets the input and decides if the user signed the phrase correctly. If the user signed it correctly, goes on to the next phrase. If the user signed the phrase incorrectly, displays the correct signing video of the word.

## How we built it

We started by downloading and preprocessing a word to ASL video dataset.
We used OpenCV to process the frames from the images and compare the user's input video's frames to the actual signing of the word. We used mediapipe to detect the hand movements and tkinter to build the front-end.

## Challenges we ran into

We definitely had a lot of challenges, from downloading compatible packages, incorporating models, and creating a working front-end to display our model.

## Accomplishments that we're proud of

We are so proud that we actually managed to build and submit something. We couldn't build what we had in mind when we started, but we have a working demo which can serve as the first step towards the goal of this project. We had times where we thought we weren't going to be able to submit anything at all, but we pushed through and now are proud that we didn't give up and have a working template.

## What we learned

While working on our project, we learned a lot of things, ranging from ASL grammar to how to incorporate different models to fit our needs.

## What's next for SignLingo

Right now, SignLingo is far away from what we imagined, so the next step would definitely be to take it to the level we first imagined. This will include making our model be able to detect more phrases to a greater accuracy, and improving the design.
## Inspiration

At the beginning of the weekend, we tossed around ideas about different apps. We got bored at some point and started discussing family. A lot of our family members seemed to have some sort of hearing loss. At this point, we started to formulate ideas around this topic. We realized that a lot of apps existed to assist individuals with hearing loss. What if we could combine a bunch of different applications into one?

## What it does

The app requires the user to create an account before doing anything. When logged in, the user is introduced to three separate features.

1. Sound Classification Page - Here the user can record sound or upload a recording. The website reports back to the user and tells them what might have made that sound.
2. Storytelling - Alright, this feature is just for entertainment! Based on the last few sounds that were detected and stored in MongoDB Atlas, we have GPT4o generate a story under a certain theme that the user selects. Themes include: funny, serious, and sad.
3. Notetaking - For deaf students, taking notes during lectures could be quite difficult. This feature allows the student to instead record the entire lecture and come up with genuinely useful notes in bulleted form.

## How we built it

We used NextJS in Javascript to build the UI of the website. The backend of the website was managed by FastAPI server build with Python. The frontend made API calls to the backend for signing/register the user and for the audio classification model (YAMNet). For the language models and speech to text, the frontend directly OpenAI API. All data including usernames, passwords, audio recordings, and audio classifications were all stored in MongoDB Atlas. We connected to the MongoDB database through PyMongo on the FastAPI server.

## Challenges we ran into

The biggest challenge we ran into had to do with handling the database. Due to finicky internet connections and relative inexperience with MongoDB Atlas (we're used to Firebase), it took us a while to configure the database. Working through these bugs took countless hours of reading documentation and hacking. But when we did, it was a HUGE relief.

While difficult at first, we managed to configure communication between the OpenAI API and our own website.

## Accomplishments that we're proud of

We all had experience in different parts of software engineering. Seeing the frontend, backend, AI, and numerous moving parts come together as one seamless web application is something that we're really proud about.

The goal of this project was to engineering something genuinely useful – something that people would actually use. While we can't say for sure, we think this hack could improve the quality of life for many individuals suffering from hearing loss. That includes our family members.

## What we learned

Of course, it took A LOT of learning to make this app. We used technologies that we barely understood just two days ago and put together and fully functional app.

It's not just that, though. It took understanding the problems that millions of people face everyday and trying to accommodate them.

## What's next for WhisperWave

Again, WhisperWave is meant to be a "toolkit for the deaf." In practice, this means we keep adding to the website to assist deaf people in their day to day lives. It is our hope to continually add more and more features including sign-language translation, emotion detection, and even voice controls.

Additionally, to make this website more accessible, we want to transform this website into an mobile app. Although the website works seamlessly on mobile browsers, the ability to have a app would greatly enhance the accessibility and personalization of our technology.