## Inspiration

We noticed that many people feel anxious about starting therapy or don't know where to get started. We wanted to create something that could naturally converse with the user in a comfortable way that could ease them into therapy.

## What it does

The Therabox is capable of having a natural conversation in a therapeutic setting; this means it requires no typing and will specifically respond to your concerns brought up. It works best in a Q&A session, and you can converse with it as you would a friend.

## How we built it

Our project has three main components: `GPT-3`, `Assembly.Ai`, and `Google Cloud`. When a user speaks to Therabox, it will detect a question and it will then keeps listening until the user finishes talking about their concerns/questions. First, we use `Assembly.Ai` to turn that voice input into text. Then we take that question and feed it into a GPT-3 model that we've fine-tuned to get a therapist-esque response. During the process, the question and response are saved into a chat log file so GPT-3 will remember prior conversations. Lastly, after saving the chat log, we use Google Cloud text-to-speech to make Therabox "talk" out loud with the user. This back-and-forth is repeated indefinitely until the user decides to end the conversation (or we run out of API credits).

## Challenges we ran into

While building Therabox we learned just how challenging it can be to integrate with some APIs. The Google Cloud API values security incredibly highly so it took a lot of effort to properly authenticate and integrate the service into our script. But once everything was set up, the integration was solid.

Interacting with the microphone and speaker connected to the Raspberry Pi also proved to be challenging. The raw data received from microphones is tough to process, so we spend a great deal of time converting it to a standard format.

## Accomplishments that we're proud of

Since GPT-3 is a general language synthesis AI, it can generate any sort of text in any context, and from any perspective. Getting its parameters just right to reliably produce realistic and helpful therapist-like responses was challenging but we’re very proud of what we achieved. Our script almost always nails the context provided by the user and gives helpful, creative advice.

Coming into the project we were concerned we wouldn’t be able to achieve a natural back-and-forth conversation with our bot since we didn’t know if we would be able to reliably detect when the user begins and finishes speaking. However, we managed with an algorithm that works with AssemblyAI’s stream endpoint to very reliably detect the beginning and end of speech without too much delay. We’re very proud of the result.

## What we learned

We learned more about working with a partially trained model when using GPT-3. Although it's already trained as a natural language processing model, we needed to add additional data to make it specifically acclimated to therapeutic conversations. Besides working with the model, we gained experience with both text-to-voice and voice-to-text translations. After completing the basic program, we ended up learning about the hardware limitations of different devices. Since we wanted the final product to run independently, we needed the software to match the Rasp-Pi hardware limitations.

## What's next for Therabox

We have big plans for Therabox in the future.

First of all, we believe that we can improve how "coherent" the responses from GPT-3 are. One of the ways we hope to improve this model in the future is to use Part of Speech Tagging (POS Tagging), which is the process of separating a sentence, attaching a tag to each word (such as articles, nouns, adjectives, etc.) and use do an analysis on the generated tags to do these following tasks:

1. Sentiment analysis: try to gauge the user's emotions to build a better prompt for GPT-3 that's more customizable to the nuance of conversation
2. Correcting input and output: since humans might not always speak in the correct grammar, correct it. GPT-3 outputs weird stuff sometimes, correct it, and help generate better output for the user.

One other way that would dramatically improve Therabox is to have therapist recommendations. Therabox is in no way a replacement for a therapist, and we want to make sure we can provide our users with the help that they need. It would be amazing to have access to a database or some sites that have information on nearby therapists.
## Inspiration

Our main inspiration was that there are quite a few times when people can't access their nearby resources when they need it the most. People have to wait for a fair amount of time just to get the basic needs of mental health. And that is when our app comes into play.

## What it does

Therapy ai is an online chatbot that gives people instant access to different medical needs. What sets therapy ai unique is that it can identify the user’s emotions and answers based on their emotion, it will give a helpful solution or advice to the user. For instance, if the severity of the user is critical, it will give the user access to their nearest health resource or connect them with a health expert. On the other hand, if the severity of the user is not so critical, it will give the user advice on what to do in the situation. And this really helps a user in difficult situations as they need help or advice on what to do.

## How we built it

We used MERN stack (MongoDB, Express, React, Node.js) to build the web application and co:here to implement the AI chatbot feature.

## Challenges we ran into

The main challenge that we ran into was trying to use co:here in our application since it was our first time using it. This was also our first time implementing a third-party API with our own custom API. Fortunately, by reading the documentation and asking questions to the co:here engineers, we were able to achieve what we wanted.

Another challenge we ran into was trying to merge the frontend and backend together.

## Accomplishments that we're proud of

We’re proud that we managed to learn and implement a technology we’ve never touched before and got it to co-exist with a working application in essentially 24 hours (since we didn’t work on Friday).

## What we learned

We learned a lot of new things within these 36 hours such as time management, communication, and leadership skills. And more importantly, we learned about the MERN Stack and using co:here API to make an automated chat bot.

## What's next for TherapyAI

We want to improve the precision and accuracy of the response that the AI provides. In order to do this, we were considering further training the AI in two ways.
Receive user feedback to check if the perceived emotion was correct and if the response was helpful
Classify invalid user input (like gibberish) and handle it accordingly
Use conversations from users to further train the AI
## **Inspiration:**

Our inspiration stemmed from the realization that the pinnacle of innovation occurs at the intersection of deep curiosity and an expansive space to explore one's imagination. Recognizing the barriers faced by learners—particularly their inability to gain real-time, personalized, and contextualized support—we envisioned a solution that would empower anyone, anywhere to seamlessly pursue their inherent curiosity and desire to learn.

## **What it does:**

Our platform is a revolutionary step forward in the realm of AI-assisted learning. It integrates advanced AI technologies with intuitive human-computer interactions to enhance the context a generative AI model can work within. By analyzing screen content—be it text, graphics, or diagrams—and amalgamating it with the user's audio explanation, our platform grasps a nuanced understanding of the user's specific pain points. Imagine a learner pointing at a perplexing diagram while voicing out their doubts; our system swiftly responds by offering immediate clarifications, both verbally and with on-screen annotations.

## **How we built it**:

We architected a Flask-based backend, creating RESTful APIs to seamlessly interface with user input and machine learning models. Integration of Google's Speech-to-Text enabled the transcription of users' learning preferences, and the incorporation of the Mathpix API facilitated image content extraction. Harnessing the prowess of the GPT-4 model, we've been able to produce contextually rich textual and audio feedback based on captured screen content and stored user data. For frontend fluidity, audio responses were encoded into base64 format, ensuring efficient playback without unnecessary re-renders.

## **Challenges we ran into**:

Scaling the model to accommodate diverse learning scenarios, especially in the broad fields of maths and chemistry, was a notable challenge. Ensuring the accuracy of content extraction and effectively translating that into meaningful AI feedback required meticulous fine-tuning.

## **Accomplishments that we're proud of**:

Successfully building a digital platform that not only deciphers image and audio content but also produces high utility, real-time feedback stands out as a paramount achievement. This platform has the potential to revolutionize how learners interact with digital content, breaking down barriers of confusion in real-time. One of the aspects of our implementation that separates us from other approaches is that we allow the user to perform ICL (In Context Learning), a feature that not many large language models don't allow the user to do seamlessly.

## **What we learned**:

We learned the immense value of integrating multiple AI technologies for a holistic user experience. The project also reinforced the importance of continuous feedback loops in learning and the transformative potential of merging generative AI models with real-time user input.