## Inspiration

Our team members have all witnessed the struggles our grandparents face due to language barriers in healthcare. They constantly need to call our parents to translate whenever a doctor or nurse visits their room. When our parents are unavailable, these challenges become even more daunting. This inspired us to create Care Voice, a solution to ensure seamless communication for non-English speaking patients.

## What it does

CareVoice is a medical translator mobile app designed to bridge the communication gap between doctors and patients. It records and translates the doctor's explanations into language that is easily understandable by the patient. Additionally, it provides background information on the doctor's recommended procedures.

CareVoice tailors its services to each user by considering their personal information, such as age, primary languages or dialects spoken, health conditions, and disabilities. This allows the app to customize translations and features to enhance accessibility. For example, it can enlarge text, offer read-aloud options, and use multisensory input to cater to various patient needs. CareVoice ensures that all patients, regardless of their disabilities, can receive and understand their medical information clearly.

## How we built it

We designed our application using Figma and implemented the front end in Swift using SwiftUI in XCode. For the backend, we integrated several advanced technologies: DeepL, which provides context-sensitive translations. Groq which was used to integrate Whisper AI which transcribes the doctor's voice into English text, as well as Meta LLaMA 70B, which offers additional information on medical terms and context, and provides personalized recommendations based on the patient's medical history. Lastly, Eleven Labs, offered various voice options for a richer user experience. This combination allows us to deliver a seamless, user-friendly, and highly personalized application.

## Challenges we ran into

We encountered several challenges while integrating the backend with the frontend, particularly in connecting our text-to-voice feature with the existing frontend components. Additionally, since it was our first time working with backend development, we faced a steep learning curve in setting up and managing the backend infrastructure.

## Accomplishments that we're proud of

Despite our initial lack of experience with mobile application development, we successfully implemented a functional backend and developed a text-to-speech feature. We are also proud of our ability to design and implement the app's user interface within the given time constraints.

## What we learned

Throughout this project, we gained valuable insights into creating a text-to-voice, and voice to text feature, implementing our design using Swift, and effectively connecting the front end with the back end. These experiences have significantly enhanced our skills and understanding of mobile app development.

## What's next for Care Voice:

Due to time constraints, we couldn't implement all the features we envisioned. In the future, we plan to add text analysis capabilities, and the ability to save data for reviewing past results and analyzed information. These enhancements will further improve CareVoice's utility and user experience.
## Inspiration

This project was inspired by Leon's father's first-hand experience with a lack of electronic medical records and realizing the need for more accessible patient experience.

## What it does

The system stores patients' medical records. it also allows patients to fill out medical forms using their voice, as well as electronically sign using their voice as well. Our theme while building it was accessibility, hence the voice control integration, simple and easy to understand UI, and big, bold colours.

## How I built it

The front end is built on react-native, while the background is built in node.js using MongoDB Atlas as our database. For our speed to text processing, we used the Google Cloud Platform. We also used Twilio for our SMS reminder component.

## Challenges We ran into

There are three distinct challenges that we ran into. The first was trying to get Twilio to function correctly within the app. We were trying to use it on the frontend but due to the nature of react native, and some node.js libraries that were being used, it was not working. We solved the problem by deploying to a Heroku serving and making REST calls.

A second challenge was trying to get the database queries to work from our backend. Although everything seems right it still did not work but to do attention to detail, and going over code multiple times, the mistake was spotted and corrected.

The third and likely biggest challenge we faced was getting the speech to text streaming input to co-operate. In the beginning, it did not stop recording at the correct times and would capture a lot of noise from the background. This problem was eventually solved by redoing it by following a tutorial online.

## Accomplishments that I'm proud of

**WE FINISHED!** We honestly did not expect to finish if you asked us at 10 pm on Saturday night. However, things came through well which we were really proud of. We are also really proud of our UI/UX and think it is a very sleek and clean design. Two other things include accurate speech to text processing and dynamically filled values through our database at runtime.

## What I learned

**Joshua** - How to write server-side Javascript using node.js

**Leon** - Twilio

**Joy** - Speech to text streaming with react native

**Kevin** - React-native

## What's next for MediSign

If we were to continue to work on this project, we would first start by dynamically filling all values through our database. We would then focus a lot of attention on security as medical records are sensitive info. Thirdly, we would upgrade the UI/UX to be even better than before.
## Inspiration

Partially inspired by the Smart Cities track, we wanted our app to have the direct utility of ordering food, while still being fun to interact with. We aimed to combine convenience with entertainment, making the experience more enjoyable than your typical drive-through order.

## What it does

You interact using only your voice. The app automatically detects when you start and stop talking, uses AI to transcribe what you say, figures out the food items (with modifications) you want to order, and adds them to your current order. It even handles details like size and flavor preferences. The AI then generates text-to-speech audio, which is played back to confirm your order in a humorous, engaging way. There is absolutely zero set-up or management necessary, as the program will completely ignore all background noises and conversation. Even then, it will still take your order with staggering precision.

## How we built it

The frontend of the app is built with React and TypeScript, while the backend uses Flask and Python. We containerized the app using Docker and deployed it using Defang. The design of the menu is also done in Canva with a dash of Harvard colors.

## Challenges we ran into

One major challenge was getting the different parts of the app—frontend, backend, and AI—to communicate effectively. From media file conversions to AI prompt engineering, we worked through each of the problems together. We struggled particularly with maintaining smooth communication once the app was deployed. Additionally, fine-tuning the AI to accurately extract order information from voice inputs while keeping the interaction natural was a big hurdle.

## Accomplishments that we're proud of

We're proud of building a fully functioning product that successfully integrates all the features we envisioned. We also managed to deploy the app, which was a huge achievement given the complexity of the project. Completing our initial feature set within the hackathon timeframe was a key success for us. Trying to work with Python data type was difficult to manage, and we were proud to navigate around that. We are also extremely proud to meet a bunch of new people and tackle new challenges that we were not previously comfortable with.

## What we learned

We honed our skills in React, TypeScript, Flask, and Python, especially in how to make these technologies work together. We also learned how to containerize and deploy applications using Docker and Docker Compose, as well as how to use Defang for cloud deployment.

## What's next for Harvard Burger

Moving forward, we want to add a business-facing interface, where restaurant staff would be able to view and fulfill customer orders. There will also be individual kiosk devices to handle order inputs. These features would allow *Harvard Burger* to move from a demo to a fully functional app that restaurants could actually use. Lastly, we can sell the product by designing marketing strategies for fast food chains.