## Inspiration

An unfortunate yet common reality for patients with memory loss (whether from Alzheimer's, dementia, amnesia, etc.) is the pain of forgetting the faces of the people in their lives—especially the ones that they love. Our goal was to use the power of AR and AI to help patients recognize faces, and to also track their memory over time. That's why we sought to augment their *recognition* with **Recogvision**.

## What it does

Recogvision is an augmented reality view for patients with memory loss that helps them remember and keep track of the people they meet. When meeting someone for the first time, our smart voice assistant picks up on their conversation and uses natural language processing to infer the person's name. Their name, along with a photo of them from our camera are stored into our database. Later, if the patient sees this person again, they are given the option to display their name on the screen. Hopefully, the patient remembers the person's name. But if they forget, they can click a button to view the name. Recogvision also tracks the number of times a patient is pressing this button—or the number of times they forget a name. This statistic over time provides valuable insight into the progression of the patient's memory, and it can be reported to medical professionals as an expedited report.

## How we built it

We built the frontend of our cross-platform application in Flutter. This app communicates with our Flask server hosted on Google Cloud, which stores and retrieves information from our Firebase database, as well as our very own person identification library we made in python.

## Challenges we ran into

We ran into A LOT of issues while working with Flutter, especially related to cropping images from the video stream in real time. We also had some cacheing difficulties with Google Cloud that we managed to work out.

## Accomplishments that we're proud of

We're really proud of our voice assistant and our person identification! These are no doubt the two most technically complex aspects of our project, and we are proud that we were able to use such high-level technology to create an intuitive, accessible, and functional solution for people who need it.

## What we learned

We learned a lot in this project! Sohit and Anthony learned about video processing in Flutter, Sarthak learned about creating a computer vision library, and Dean learned about building backend servers and databases.

## What's next for Recogvision

We want to take this idea further! We think this is a very effective way to help patients with memory loss, and we want to incorporate more features into the app such as showing the person's relation to you, information about them, or details about when you first met them.
## Inspiration

The inspiration for this project came to us from recognizing that this simple at the first look app could have a great social cause, and there is nothing like that on the market. We wanted to create a tool for Alzheimer’s patients that would be a single platform for aggregates tools for their individual needs
We also wanted to get experience and play with different technologies.

## What it does

Our app helps Alzheimer’s patients recognize people, associated memories, and manage reminders for daily activities using Augmented Reality and Machine Learning. Our reminder system helps users to keep track of their daily routines and medications.

## How we built it

Our iOS application uses Microsoft Azure’s Custom Vision to recognize family and friends. The app uses coreML, the ARKit, and the Vision framework to label the recognized people in real time. Using the Houndify API custom commands and the Oracle Cloud Database, the user can verbally request and receive information about people saved to their account. We also have a voice assistant utilizing Almond Voice API offered by Stanford to greet the user.

## Challenges we ran into

During the project, we ran into a few challenges -- starting with lack of documentations for some API’s and finishing with software failure (iPhone simulator)
We used several powerful API’s, yet it was a challenge to implement them to our application due to the lack of good documentation. In addition, all of us were pioneers in using RESTful API’s.

## Accomplishments that we're proud of

We proud that out application has great social cause. We hope, it may help the patients to ease their daily life. We are proud that we were able to overcome implementing difficulties.

## What we learned

We learned how to implement different API’s such as voice assistance and computer vision.

## What's next for memory lane

We need to provide users with the option of uploading images and improving accuracy of recognition.
## Inspiration

**According to the Alzheimer’s Association, 6.7 million Americans age 65 and older are living with Alzheimer's in 2023.** Alzheimer’s is a gradually progressive brain disorder involving memory loss. It is the most common form of dementia; people forget who their loved ones are and cannot carry out daily tasks anymore. Alzheimer’s and dementia is not only a personal health crisis but impacts family, friends, and caregivers. It is important to focus on the prevention and slow progression of symptoms related to memory loss. **How can we use the Memory Palace – a psychology-based technique where people can associate mnemonic images in their mind to places they know – to help prevent and ease the lives of those with Alzheimer’s and dementia?**

There is a lot of psychology research focused on memory and learning that can be used to guide technical applications and enhance memory performance. Our team is very interested in memory and learning mechanisms, which have inspired our idea to use scientific background to improve memory.

## What it does

**Memory Playground is a web application that helps boost memory recall and retention through the Memory Palace technique, especially for senior citizens and those with Alzheimer’s and dementia.** The application allows users to pick a setting/environment and list out words that are related. Then, we create broad yet distinct categories for the words. From here, we have users practice classifying objects, allowing them to create visual mappings of images physically and mentally. This allows them to strengthen memory connections and enhance memory performance. We also give them other words that fall into those categories to expand on the established mental connections.

Memory Playground also uses an integration of zero-knowledge proofs. It stores uploaded data securely on servers and allows users to anonymously interact with the application without revealing personally identifiable information, which is ideal for those concerned about privacy.

## How we built it

We used the OpenAI API to prompt their GPT-4 model for category groupings and new objects. Then we used Together.AI's stable diffusion model for image generation. From there we connected the Python components to the web side using Fetch API. We built the web application with React, HTML, CSS, and JS. We used Flask to integrate the Python backend with the frontend.

## Challenges we ran into

1) Working with multiple servers
2) Integrating Flask with React
3) Learning to use multiple APIs to integrate various services
4) Implementing drag and drop functionality using use states
5) Limited credit and GPU usage
6) Utilizing multimodal machine learning models

## Accomplishments that we're proud of

We are proud of how we efficiently and quickly were able to understand and implement new technologies and concepts. We were new to Flask and a lot of the recent AI technology. We are also proud of how we worked as a team, ideation stages to product creation. Additionally, we are proud of how we were able to integrate multiple varying technologies.

## What we learned

We developed a lot of technical skills involving using APIs, prompt engineering, model optimization, Flask, managing multiple server applications, and web development. We also learned a lot about the practical applications of AI in healthcare towards a potential treatment of Alzheimer's and dementia. It is critical for us as a society to consider how we can *prevent*, not just treat, such diseases.

We also learned a lot about the engineering design process. Throughout the hackathon we went through research, ideation, designing, prototyping, and building phases. We gained a lot of skills through this process that helped us as we adapted to new technologies and grew our knowledge base.

## What's next for Memory Playground

1) Scaling: We are looking to make our application public and available to the community. This would involve cloud data storage (ex. Firestore) and increased efficiency to manage larger requests.
2) Speech-to-text recognition: We would like to implement a speech to text recognition model so that we can utilize verbal connections and improve accessibility.