## Inspiration

According to the National Assessment for Educational Progress (NAEP), ⅔ of fourth grade students are not meeting a proficient reading level. This has been the result of a combination of reduced public education funding, the impact of the COVID-19 pandemic and online earning, and the ongoing debate about the use of phonics vs. whole language for literary education. After talking to a teacher from Seattle, we identified some firsthand challenges that educators face due to this decline – a strong need for improvement in reading comprehension for proficiency in other subjects was labeled as a top concern.

However, we thought that it was still important to continuously integrate technology in students’ lives, especially with many classrooms already adopting tablets and computers in class. Thus, why not take advantage of the tools of the future? With these thoughts in mind, we created Voba – a tool to make learning vocabulary fun via a choose-your-own adventure game.

## What it does

Voba is an educational tool that allows students to create their own stories based off of teacher-assigned or self-added vocabulary words. By allowing students to directly engage with storytelling, we allow them to engage more directly with the vocabulary and make every experience different. With the number of possibilities that come with every direction a story could take, it allows for unlimited fun!

Voba supports 3 different learning modes:

1. **Creative Mode**: This mode is a free-for-all story generation party, where the AI agent generates a starter story based on the vocabulary checklist and multiple options to continue the story with sentences that utilize the vocabulary in the list. This encourages students to actively and fearlessly try using new words while introducing unfamiliar vocabulary through the way they were meant to be used.
2. **Test Mode**: This mode employs a multiple-choice system where instead of generating sentences which ALL work, the agent will come up with 3 incorrect uses of the vocabulary and 1 correct use. If the student gets it wrong, the companion (Vovo) will help them understand why the answer was incorrect.
3. **Mixed Mode**: This mode combines elements of both Creative and Mixed mode. Here, we have 2 correct and 2 incorrect answers, which allows students both the ability to explore but also to understand the use of vocabulary through varied contexts.

In addition to this main learning mode, we also incorporated the following to support learning and information retention:

1. **Highlighted Key Words + Definitions**: In every generated sentence, the words on the vocabulary list will be highlighted. When the student hovers over it, it will show the definition of the word, helping them understand and process how it is used in that context.
2. **Audio**: Utilizing OpenAI’s text-to-speech model, we implemented audio for every sentence so students can also hear how the word is used in the sentence. This reinforced another pillar of learning vocabulary – the auditory versions. It can also help them grasp pronunciation and tone.
3. **Vocabulary List View:** We implemented both a ‘student’ and ‘teacher’ mode. If a student simply wants to review vocabulary words, then they can view the “vocabulary list” from their home page and click through it to see that and a definition of the word. If they click on the word, then it will say the pronunciation of the word out loud. If you are entered as a teacher, then you can add in words.
4. **Vocabulary Word Review Tracking**: When playing the game, the application and the AI agent will keep track of which words the student has reviewed and how often, providing both the student and the teacher (for progress tracking purposes) a way to see their progress visually. The AI Agent can also use this data to know which words to review, allowing it to focus on words that are in-need of review and bring up words that haven’t been used in a while.
5. **Mascot Companion**: For each mode, we have a mascot companion to accompany the student on their adventure! Vovo the Elephant provides encouragement upon correct answers, completion of the story, and provides feedback about incorrect answers in Quiz and Mixed mode. Having a cute mascot helps put the students at ease and become more engaged in the story.
6. **Advanced Sentence Generation**: For every sentence generated, the agent can also use 1-2 words, showing the students how multiple difficult vocabulary words can be combined in a sentence.

With a combination of these features, it creates a fun, seamless experience for students to learn new words and reinforce the learning. It also offers a unique take on the creative process with story generation and AI as well, as it also encourages students to learn the storytelling process!

## How we built it

**Frontend:** React, TailwindCSS

**AI:** OpenAI Text-to-Speech, OpenAI GPT-4, LangChain

**API:** FastAPI, AWS API Gateway

**Infrastructure / Management:** AWS Lambda, AWS S3, Firebase hosting

**Design:** Canva

## Challenges we ran into

**Front-end and back-end integration**: It was difficult to piece together the infrastructure, frontend, and backend components, especially with all of us working on different parts at the same time.

**Passing and Serving Audio Files via. API:** Since none of us had ever worked with text-to-speech AIs, it was difficult to figure out the best way to ensure that the entire sentence generated got to the audio files and then sent back to the app.

**Prompt Generation:** It’s difficult to get the correct use of the word according to the given definition, as many words have many different definitions! On top of this, we had to ensure that the AI didn’t come up with words that were too complex for a 2nd grader to understand. Lastly, we had to ensure that the story would follow a kind of “storytelling structure,” with an introduction, climax, conclusion, but still be interesting instead of formulaic.

## Accomplishments that we're proud of

Two of our group members were first-time hackathon participants, so we’re really proud of how much everyone was able to contribute to both the brainstorming and implementation of the project. Each member of the team took charge of a specific part and learned something new!

1. **TLDR**; we’re super proud that we built a full-stack project with multiple APIs!
2. **LangChain**: None of us had ever used LangChain before, so the majority of our debugging involved LangChain issues and how we maintained the use of the vocabulary words, frequency of word use, keeping it at the appropriate level, and more.
3. **Child friendly UI design**: We felt that we were able to effectively capture this! In our research, we saw that a lot of child-focused websites (i.e. National Geographic Kids, PBS Kids, etc) were very sloppily designed. We wanted to strike the balance between fun and bold but also structured.
4. **Fun!:** All of us were really pleasantly surprised about how fun this entire app is, even for us as college students. It brings back a kind of nostalgia for childrens’ stories while being genuinely engaging enough for us to play around with it for hours without fail (We coincidentally got so many pirate stories. So many.)

We pushed ourselves a lot out of our comfort zone with the scope of the project. Most of us only have ever had some front-end experience, so creating a full-stack project was extremely new to us, especially with 2 new hackers!

## What we learned

1. How to design for a specific audience – building for elementary school students is much different than building for high school / college students. A lot of the coding, colors, design, and prompt engineering had to keep our young users in mind.
2. Everyone on the team learned a new technology, whether it’s on the back-end, front-end, or AI / infrastructure side.
3. Effective communication and prioritization of tasks – we were extremely ambitious and had to scale and alter the idea as it developed further and we began programming.

## What's next for Voba

Looking ahead, we envision Voba evolving into a more comprehensive tool for students to have fun and learn vocabulary on their own and as a tool for instructors to use.

**Teacher Portal**: A dedicated dashboard for teachers to manage classrooms, track class and student progress, assign grades, and offer reward incentives.

**Expanded Learning Models**: We could introduce more game modes that test different things, for instance, a pure quiz mode or a ‘kid’ mode.

**Multiple-meaning words**: Include multiple definitions for each word

**UI/UX Enhancements for Accessibility**: Implementing research-backed strategies to help students learn, especially those with learning impediments, disabilities and more. This could involve:

* Expanding the audio function for blind and visually-impaired students
* Integration of illustrations for a more engaging story
* Color-blind mode, slower paced assignments, difficulty adjustments
* Progress Tracker + Badges: To encourage students to play, we could implement some more incentives that either a teacher could implement or a student could obtain on their own time.
Saving Stories: After students have created their stories, we’d love to implement an export function to allow them to continue building on them or to share it with their friends and family!
## Inspiration

Our innovative platform combines the power of conversational AI with the fun of interactive games, transforming traditional language learning into an engaging and playful experience. With a vibrant, child-friendly interface and exciting games like 'Fruit Slash' for vocabulary building and a math challenge that lets kids solve basic arithmetic problems in Spanish, we make learning feel like play.
The magic lies in our use of voice technology.

## What it does

Kids respond to challenges by speaking, and our integration with Gladia’s speech-to-text engine ensures that their responses are instantly processed and validated. This conversation-based approach allows young learners to practice speaking Spanish in a natural and fun way, making language learning more interactive than ever.
We believe that the best learning happens when you're having fun, and we’re changing the way kids learn languages—one conversation at a time.

## How we built it

• Design & Prototyping: Used Figma to create a fun and interactive user interface, focusing on a kid-friendly design.
• Frontend Development: Built the frontend using React and Next.js, implementing responsive components and game interfaces.
• Speech-to-Text Integration: Integrated Gladia for live audio processing, converting speech into text in real time.
• Text Processing Logic: Implemented simpdjson to handle JSON queries and manage data from the speech-to-text responses.
• Game Functionality: Developed game UIs that utilize the processed text to validate responses and provide feedback within interactive games.

## Challenges we ran into

• Limited Time and Expertise: Couldn't fine-tune voice models to the desired accuracy due to time constraints and lack of expertise.
• Latency Issues: Experienced delays with API calls between the speech-to-text service and the conversational agent, affecting response times.
• Poor Internet Connectivity: Encountered frequent connectivity issues throughout the hackathon, costing valuable development time.
• Limited Workspace: Faced difficulties collaborating effectively due to cramped working conditions and lack of sufficient space.

## Accomplishments that we're proud of

• Team Collaboration: Successfully worked together despite various struggles, demonstrating strong teamwork and resilience.
• Rapid Prototyping: Quickly tested different voice software to validate the idea and identify a feasible solution.
• UI Design and User Experience: Created a friendly, appealing, and fun interface that makes learning enjoyable for kids.
• Clean and Interactive Interface: Developed a clean, user-friendly interface that enhances the learning experience through engaging interactions.

## What we learned

• Real time streaming ain't easy, if you want low latency! We promise awesome latency streaming!

## What's next for Funlingo

• Character-Based Conversation Agents: Train voice models to incorporate popular TV and movie characters, allowing kids to learn from their favorite personalities and making the experience more engaging.
• Personalized Pronunciation Tutor: Develop a conversation agent that listens to kids’ pronunciation and provides real-time feedback, acting as a personalized tutor to help them improve their speaking skills.

• Research-Backed Educational Content: Collaborate with educators and researchers to design games that are not only fun but also scientifically proven to enhance language learning outcomes.
# Healora

## Background

**Why should nurses and doctors have to rely solely on manual inputs when modern healthcare technology has so much more potential?**

In hospitals and healthcare settings, **medical professionals are often overwhelmed**, managing complex workflows, multiple patients, and endless data inputs, often missing key details that could improve patient care. But what if **technology could alleviate this burden**, helping healthcare workers make quicker, more accurate decisions?

In today’s world, we have **empathetic AI, predictive models**, and **vast amounts of data** at our disposal. Yet, many hospital systems continue to rely on **outdated methods** for managing symptoms, diagnoses, and patient interactions. **Healthcare workers lose valuable time** manually entering and interpreting data when **AI could be working alongside them**.

**Healora was created to bridge that gap**. Empowering medical staff with **AI-driven tools**, Helora leverages **Nurse Joy**, a virtual assistant, to intelligently **track symptoms**, perform **predictive analysis**, and provide **emotional understanding** in patient interactions. With Helora, hospitals can **streamline workflows**, allowing nurses and doctors to focus more on what truly matters—**saving lives and improving patient outcomes**.

![image](https://github.com/user-attachments/assets/740d714d-f020-4743-9f51-25ea500a2408)

## What is Healora?

Healora is an AI-powered healthcare platform designed to streamline patient care by integrating real-time health monitoring, virtual assistance, and efficient data management for healthcare providers.

1. The pre-screening page allows patients to enter their initial health data and symptoms, which helps healthcare providers assess their condition quickly before moving forward with more detailed examinations.
![image](https://github.com/user-attachments/assets/d9a38dbd-0ef4-44c3-8806-24692db0a2db)
2. The treatment page is where Nurse Joy, Healora's AI assistant, interacts with patients. This page also displays vital signs, such as respiratory rate, blood pressure, and heart rate, allowing medical staff to monitor patient health in real-time.
![image](https://github.com/user-attachments/assets/eec45b75-70cf-4ab3-81ea-4727d2feb651)
3. The patient list provides healthcare professionals with an overview of all patients currently under care, including their basic health details and treatment status.
![image](https://github.com/user-attachments/assets/220c5667-9c20-4c53-9cf8-0ca0fff03ed2)
4. The staff table offers a detailed view of the healthcare team, enabling easy management of roles and responsibilities within the medical staff.
![image](https://github.com/user-attachments/assets/cc24c6a6-f2cf-40ac-b9bf-73257b0588c3)

## Features

* **Symptom Tracking**:
Nurse Joy, Helora's virtual assistant, allows patients to log symptoms efficiently. These symptoms are recorded and analyzed to help healthcare workers make data-driven decisions.
* **Predictive Analysis**
Helora uses AI models to perform predictive analysis on the collected data, offering potential diagnoses and next steps for treatment based on the symptoms entered.
* **Agentic Backend**
Helora uses AI agents for majority of its backend tasks, powered by Fetch.AI
* **Empathetic AI**:
Helora incorporates emotional intelligence to provide compassionate, context-aware responses, ensuring that patients feel understood and supported during their care, powered by Hume AI
* **AI-Generated Feedback**:
Nurse Joy provides real-time feedbackafter each interaction, helping healthcare professionals review patient information, track symptoms, and make informed decisions quickly.
* **Real-Time Data Analysis**
Helora processes patient data in real-time, providing medical staff with up-to-date information on patient vitals and symptoms.
* **Voice** Interaction
Helora supports voice input, giving patients flexible communication options based on their preferences.

## Planning

We began by thoroughly researching the current state of hospital workflows, particularly focusing on how AI could alleviate the pressures faced by healthcare workers. From understanding existing tools to studying user needs, our research laid the foundation for building Helora.

Once we had a clear understanding of the problem space, we created user personas and developed user flows to guide our design. Using Figma, we designed prototypes to rapidly iterate and refine the user experience. Our focus was on creating a system that is intuitive for both healthcare workers and patients, with a clean, user-friendly interface that reflects the needs of medical professionals.
![image](https://github.com/user-attachments/assets/b34917e6-f38f-4ce0-8895-d0250b9bea22)

## System Architecture

![image](https://github.com/user-attachments/assets/457bb7f2-bfd7-4809-bda5-1357e1cd26c0)

### Frontend

**Helora** is a responsive desktop and mobile-friendly web application built using Next.js, with Tailwind CSS for streamlined and highly customizable styling. We use shadcn components to provide a cohesive design system and Framer Motion to add smooth, engaging animations for improved user experience.

Our frontend efficiently manages real-time communication between healthcare professionals and Nurse Joy, ensuring seamless interaction. Data flows between users and the backend are handled securely, providing healthcare workers with immediate feedback from the AI and allowing for effortless tracking of symptoms, analysis, and patient interactions.

### Backend

Our project's backend is a robust and scalable architecture designed to deliver advanced AI capabilities. Our backend employs **Fetch AI agents** orchestrated through **FastAPI** to manage workflows and interactions efficiently. We use **Supabase with PostgreSQL** for real-time data storage and management. For language-based analysis, we have integrated **Llama** and **Mixtral** models via **Groq**, while **Hume AI** powers our conversational tasks with advanced sentiment and emotional analysis.

![image](https://github.com/user-attachments/assets/a570e5ab-691c-4da4-8a6d-3a511330e01d)
*The technologies that we used to power Healora.*

## To run the project

Clone the repository at [Healora](https://github.com/Zalinto/med-ai)
Create a **.env** file, place it in the root directory and fill in with api keys of your configuration:

```
NEXT_PUBLIC_SUPABASE_URL=
NEXT_PUBLIC_SUPABASE_KEY=
SESSION_SECRET=
SESSION_EXPIRES_IN=

NEXT_PUBLIC_DEVELOPMENT=
NEXT_PUBLIC_BASE_LOCAL_URL=
NEXT_PUBLIC_BASE_PROD_URL=

HUME_API_KEY=
HUME_API_SECRET=
HUME_CONFIG_ID=

NEXT_PUBLIC_HUME_CONFIG_ID_SUK=
HUME_API_KEY_SUK=
HUME_SECRET_KEY_SUK=

```

Run the following commands(terminal) in the root directory:

```
npm install
npm run dev

```

## Use cases

![image](https://github.com/user-attachments/assets/e3bfdf7f-fbfb-4439-a364-b8b95eb09f8c)

## Takeaways

### What we learned

Through building Healora, we discovered the complexities of integrating AI into healthcare workflows. We learned how crucial it is to blend advanced technology with empathy to enhance patient care. We also deepened our understanding of AI agents and language models, realizing their potential to make a real difference in medical workflow management and Nurse Assisstant settings

### Accomplishments

We’re incredibly proud of what we achieved during this hackathon. Not only did we integrate a wide range of features like symptom tracking, AI-powered predictive analysis, and real-time empathetic feedback, but we also managed to design a clean, user-friendly interface tailored for healthcare professionals. This project also gave us the opportunity to apply research in meaningful ways by reading research papers and using user personas to ensure Helora meets the specific needs of both nurses and patients.

On the design side, we’re proud of the consistent and intuitive UI built using Tailwind CSS and Framer Motion for smooth interactions. Helora’s design language prioritizes clarity and accessibility, catering to both healthcare workers and patients alike.

### What's Next

Moving forward, we plan to focus on scaling Helora’s AI capabilities. While we’ve implemented empathetic AI and predictive analysis, we see room for improvement in terms of diagnosis accuracy and real-time feedback. Additionally, we aim to expand Helora’s capabilities by incorporating more comprehensive datasets to improve symptom recognition and predictive healthcare outcomes.

We also want to refine Helora’s security and privacy features, ensuring patient data is handled with the utmost care and confidentiality. Our next steps include conducting more user testing, enhancing multilingual support, and refining Helora's user experience for different healthcare roles (nurses, doctors, HR personnel).

Helora’s journey is just beginning, and we believe it has the potential to revolutionize how healthcare professionals interact with patients, improving both workflow efficiency and patient care.