## Inspiration

Our inspiration for creating this AI project stemmed from the desire to leverage current technology to enhance the lives of our community members, particularly the elderly. Recognizing the difficulties faced by aging individuals, including our own grandparents, to embrace technological advancements, and their concerns about health conditions, we sought a way to integrate technology and healthcare services seamlessly into their lives. Thus, SageWell was conceived—an AI companion designed to provide personalized support to seniors by navigating medical information alongside their healthcare providers. At SageWell, we believe in accessibility of AI for all age groups.

## What it does

SageWell enables users to seek medical information through natural language interactions, allowing elderly individuals to pose questions verbally and receive spoken responses, mimicking human conversation.

## How we built it

SageWell leverages the capabilities of two Monster APIs: OpenAI-Whisper Large-v2 for speech-to-text conversion and Meta's llama-2-7b-chat-hf for refining our reinforcement learning model. Our model was trained on MedQuAD, a comprehensive dataset containing over 47,000 medical question-answer pairs, Drugs, Side Effects and Medical Condition dataset, and Drugs Related to Medical Conditions dataset. The frontend of the web app was built using React, a JavaScript library, and the service we integrated into our app to store all the audio files was Firebase.

## Challenges we ran into

In the process of developing SageWell, we encountered several challenges. Since there were many integrations in our application, from speech-to-text transcription and MonsterAPIs for fine-tuned LLMs, to storage service providers, we faced difficulties trying to link all the individual pieces together. We learned several new tools and technologies and also spent time fine-tuning models to provide medical information. Additionally, we encountered setbacks when we were finishing up our project as we were facing CORS errors when making API calls from the browser - we were able to work around this by adding a proxy, which served as a bridge between our client and server.

## Accomplishments that we're proud of

We empowered our primary user demographic, the elderly, to engage with SageWell through voice interactions instead of having to type text. We addressed the challenges that many seniors encounter when typing on digital devices, thereby increasing the accessibility of the process of seeking medical information and ensuring inclusivity in AI for all age groups, especially the elderly.

Our reinforcement learning model has demonstrated effectiveness, evidenced by the decrease in training loss after each iteration and to 0.756 at the end. This indicates that the model fits well on the training data.

We merged three datasets: the Medical Question Answering data set, the Drugs, Side Effects and Medical Condition data set, and the Drugs Related to Medical Conditions data set. By training our model for SageWell on these datasets obtained from NIH websites and drugs.com, we allowed SageWell to provide medical information to users based on reliable, comprehensive databases.

## What we learned

Through this project, we gained valuable insights into the startup ecosystem and developer space. We expanded our skill set by refining our model, working with new APIs and storage providers, and creating a solution that addresses the specific challenges faced by our target audience.

## What's next for SageWell

The journey of SageWell is just beginning. Moving forward, we aim to expand its capabilities to assist in additional areas crucial for the well-being of the elderly, such as medication reminders and guidance on accessing support for domestic chores. Furthermore, we envision integrating features that facilitate connections between seniors and younger generations, including their grandchildren and other youth in their communities. By fostering these intergenerational connections, SageWell will expand its targeting market size by not only keeping the elderly engaged with their loved ones but also ensuring they remain connected to the evolving world around them. Through SageWell, we look forward to continuing to push for accessibility of AI for all age groups.
## Inspiration 🌎

One of the challenges in our multicultural country is providing healthcare to non-verbal or non-English speaking patients.

Health care professionals often face a challenge when communicating with these patients. Bedside workers are not provided a translator and often rely on the patient's family for communication. This leads to miscommunication with the patient and misinformation from the family that may not be entirely truthful when answering questions involving medical history.

This also creates a problem when the nurses are trying to fully inform their patients before receiving their consent for a treatment. Nurses are liable is they don't received informed consent and communicating with non-verbal or non-English speaking patients can become increasing stressful for this reason.

Our team wanted address all of these problems while improving the efficiency and quality of life for these hard working, under staffed professionals.

## What does it do 🤔

Our application provides a system to help healthcare workers communicate with non-verbal and non-English speaking patients in a user friendly manner using real-time transcription and translating, graphical pain indicators, and visual symptom assessment surveys.

## How we built it 🔨

Using AssemblyAI + Google Translate API we created a real-time transcription and translating system that can cross the language barrier between the patient and the healthcare worker.

Interactive healthcare diagrams made using HTML and JavaScript designed to be simple and user-friendly were constructed to help patients visually communicate with professionals quickly.

All of the data is stored for later use in a secure hospital database to help keep track of patients' progress and make for easy data sharing between shifts of nurses.

## Challenges we ran into 🔎

Configuring the translations API and incorporating it into a browser environment proved to be very difficult when developing the backend. After hours of reading documentation, consulting mentors, and trying different approaches we finally got these tools to work seamlessly for our subtitle generator.

## Accomplishments that we're proud of 💪

We were able to implement real-time captioning using AssemblyAI and translation using the Google Translate API. We are also proud that we made a fully functioning web application in only 36 hours without the use of a framework. We think our program can provide real benefits to the healthcare industry.

## What we learned 🧠

We all learned how to use AssemblyAI and some of us learned JavaScript for the first time. We got to build on our UI development skills and refine our knowledge of databases.

## Looking Forward ⏩

We plan to implement foreign to English language translation to improve communication between the patient and nurse. With more time, we would have added functionality for nurses to customize the symptoms questionnaire and patient needs menu to further improve user experience.
## Inspiration

One of our teammate’s grandfathers suffers from diabetic retinopathy, which causes severe vision loss.
Looking on a broader scale, over 2.2 billion people suffer from near or distant vision impairment worldwide. After examining the issue closer, it can be confirmed that the issue disproportionately affects people over the age of 50 years old. We wanted to create a solution that would help them navigate the complex world independently.

## What it does

### Object Identification:

Utilizes advanced computer vision to identify and describe objects in the user's surroundings, providing real-time audio feedback.

### Facial Recognition:

It employs machine learning for facial recognition, enabling users to recognize and remember familiar faces, and fostering a deeper connection with their environment.

### Interactive Question Answering:

Acts as an on-demand information resource, allowing users to ask questions and receive accurate answers, covering a wide range of topics.

### Voice Commands:

Features a user-friendly voice command system accessible to all, facilitating seamless interaction with the AI assistant: Sierra.

## How we built it

* Python
* OpenCV
* GCP & Firebase
* Google Maps API, Google Pyttsx3, Google’s VERTEX AI Toolkit (removed later due to inefficiency)

## Challenges we ran into

* Slow response times with Google Products, resulting in some replacements of services (e.g. Pyttsx3 was replaced by a faster, offline nlp model from Vosk)
* Due to the hardware capabilities of our low-end laptops, there is some amount of lag and slowness in the software with average response times of 7-8 seconds.
* Due to strict security measures and product design, we faced a lack of flexibility in working with the Maps API. After working together with each other and viewing some tutorials, we learned how to integrate Google Maps into the dashboard

## Accomplishments that we're proud of

We are proud that by the end of the hacking period, we had a working prototype and software. Both of these factors were able to integrate properly. The AI assistant, Sierra, can accurately recognize faces as well as detect settings in the real world. Although there were challenges along the way, the immense effort we put in paid off.

## What we learned

* How to work with a variety of Google Cloud-based tools and how to overcome potential challenges they pose to beginner users.
* How to connect a smartphone to a laptop with a remote connection to create more opportunities for practical designs and demonstrations.
How to connect a smartphone to a laptop with a remote connection to create more opportunities for practical designs and demonstrations.
How to create docker containers to deploy google cloud-based flask applications to host our dashboard.
How to develop Firebase Cloud Functions to implement cron jobs. We tried to developed a cron job that would send alerts to the user.

## What's next for Saight

### Optimizing the Response Time

Currently, the hardware limitations of our computers create a large delay in the assistant's response times. By improving the efficiency of the models used, we can improve the user experience in fast-paced environments.

### Testing Various Materials for the Mount

The physical prototype of the mount was mainly a proof-of-concept for the idea. In the future, we can conduct research and testing on various materials to find out which ones are most preferred by users. Factors such as density, cost and durability will all play a role in this decision.