## Inspiration

2 days before flying to Hack the North, Darryl forgot his keys and spent the better part of an afternoon retracing his steps to find it- But what if there was a personal assistant that remembered everything for you? Memories should be made easier with the technologies we have today.

## What it does

A camera records you as you go about your day to day life, storing "comic book strip" panels containing images and context of what you're doing as you go about your life. When you want to remember something you can ask out loud, and it'll use Open AI's API to search through its "memories" to bring up the location, time, and your action when you lost it. This can help with knowing where you placed your keys, if you locked your door/garage, and other day to day life.

## How we built it

The React-based UI interface records using your webcam, screenshotting every second, and stopping at the 9 second mark before creating a 3x3 comic image. This was done because having static images would not give enough context for certain scenarios, and we wanted to reduce the rate of API requests per image. After generating this image, it sends this to OpenAI's turbo vision model, which then gives contextualized info about the image. This info is then posted sent to our Express.JS service hosted on Vercel, which in turn parses this data and sends it to Cloud Firestore (stored in a Firebase database). To re-access this data, the browser's built in speech recognition is utilized by us along with the SpeechSynthesis API in order to communicate back and forth with the user. The user speaks, the dialogue is converted into text and processed by Open AI, which then classifies it as either a search for an action, or an object find. It then searches through the database and speaks out loud, giving information with a naturalized response.

## Challenges we ran into

We originally planned on using a VR headset, webcam, NEST camera, or anything external with a camera, which we could attach to our bodies somehow. Unfortunately the hardware lottery didn't go our way; to combat this, we decided to make use of MacOS's continuity feature, using our iPhone camera connected to our macbook as our primary input.

## Accomplishments that we're proud of

As a two person team, we're proud of how well we were able to work together and silo our tasks so they didn't interfere with each other. Also, this was Michelle's first time working with Express.JS and Firebase, so we're proud of how fast we were able to learn!

## What we learned

We learned about OpenAI's turbo vision API capabilities, how to work together as a team, how to sleep effectively on a couch and with very little sleep.

## What's next for ReCall: Memories done for you!

We originally had a vision for people with amnesia and memory loss problems, where there would be a catalogue for the people that they've met in the past to help them as they recover. We didn't have too much context on these health problems however, and limited scope, so in the future we would like to implement a face recognition feature to help people remember their friends and family.
## Inspiration

I've been previously locked out of my own car before with no spare key or way of getting in far from home.

## What it does

Knock Lock is an alternative way to unlock your car if you don't have your keys or don't want to break into your own vehicle.

## How we built it

A Piezo speaker listens for knocks based on the number of knocks and the time interval between each knock. This is run on an Arduino R3 microcontroller which is connected to a relay in the car that is responsible for the power locks. If the sequence of knocks is correct, then the Arduino will send an electrical pulse to the relay that will unlock the doors.

## Challenges we ran into

The main issue we had with the project was designing and soldering the electrical circuit running from the car, the speaker, and Arduino. Another issue we had was wiring and working on the car in the cold.

## Accomplishments that we're proud of

We are proud of managing to get the Arduino to interact with the car based on input from the speaker.

## What we learned

As this is our first hardware based project, we learned a great deal in how electrical components work and being able to interact with a car's electrical systems on a rudimentary level.

## What's next for Knock Lock

We wish to wire the controller to the car's ignition to be able to not only unlock the car based on the correct knock, but also start the car simultaneously.
## Inspiration

Our inspiration comes from one of our team member's bedtime story sessions with his little cousin. Whenever he reads him a picture book, his cousin always has questions like, "Why did Goldilocks walk into the house?"—and most of the time, we don’t have the answers. We wished there was a way for him to ask the characters directly, to hear their side of the story. That’s where the idea for **LiveStory** came from. We wanted to bring storybook characters to life and let kids interact with them directly to get answers to their questions in real time.

## What it does

**LiveStory** is a children's storybook that comes alive! At ANY point in the story, interact with a character on the page and chat with them; how are they feeling about what just happened?
With industry-leading AI voice-to-voice pipelines, readers can feel the emotion of their favourite characters.

## How we built it

Our web application was built primarily using **Reflex**:

* Reflex for both frontend and backend
* React.js for managing the database of each page and character
* Custom assistants powered by Vapi, Groq, Deepgram, and 11Labs to simulate character interactions
* Reflex for Deploy

## Challenges we ran into

* We initially struggled with Reflex, but over time, it became our go-to tool for building the project.
* We had to prevent characters from spoiling the story by restricting their responses to what the reader had already seen. To solve this, we fed the accumulative story log into the voice API, ensuring characters only referenced the relevant parts of the story.

## Accomplishments that we're proud of

* Completing the project on time and getting it fully functional!!!
* Learning Reflex and Lottie from scratch and successfully implementing them over the weekend.
* Collaborating with amazing Reflex engineers to create a solid product based on their platform.
* Committing 20+ hours and $1,000 on travel from Waterloo, Canada, to make this hackathon happen!

## What we learned

* Making a full-stack app in Reflex
* Implementing beautiful vector animations with Lottie
* Implementing voice-to-voice models in web apps

## What's next for LiveStory

As time is the biggest limitation during a hackathon, we would have loved to pour more time into the art to make a more beautiful experience.

* More stories!
* More animations! Characters could, based on the emotions of their speech, have reactive animations
* Character Interaction Interface: a more advanced UI can note your **emotions**
* **Choose your own adventures!** With supported stories, the conversations with the characters could influence the story!
* Customization for reader! We can also try feeding reader's information such as name, hobby and academic interest to serve better user experience.