## Inspiration

Discord is used by many companies, students, or basically anyone to connect with their peers, friends, teachers, or coworkers. It's massively gained traction throughout the pandemic, now becoming an essential component of our lives. It's become so popular that even universities and schools are creating discord keys to pass on important information to students. Several people use it for hosting podcasts, company meetings, online lectures, etc. Being long-time users of discord, we felt that we could enhance everyone's experience by allowing people to record, save and get transcriptions of their voice calls. We wanted to leverage the power of AI to make do all of this effortlessly. Because the Whisper model from OpenAI supports various languages, it is more tolerant of speakers of other languages and promotes diversity online.

## What it does

Discribe is a Discord bot that records, transcribes, and summarizes your Discord voice calls and sends them over to your chat, whenever you wish.

## How we built it

None of us had made a discord bot before, but we knew of various technologies that could be used to code one. We eventually decided on discord.js since some of us had experience with JavaScript. Discord does not support recording audio in its API, but we found a solution involving streams that allowed us to record. Using ffmpeg, we were able to create mp3 files from these streams.
After that, we experimented with OpenAI’s Whisper, an automatic speech recognition system. It was simplest for us to communicate with the AI using Python, so we used that. Once we could convert any .mp3 files into text, we used Co:heres API to create an NLP model that summarises text. We then combined the two models, to create a programme that summarises the .mp3 file.
Lastly, we created a system for the JavaScript bot and Python program to speak to each other while both are running.

## Themes

**Discovery** \
Normally we rely on video recorders and play back to store our calls, and this doesn't work since a voice recording will always have random junk in it that we don't want to see. If you've ever put a lecture recording on double speed to save yourself the boredom, you will know what we mean. New technologies such as Whisper are just becoming stronger and stronger at converting the voice into text, that it makes sense to apply them to a situation like this. Combined with Cohere's powerful language processing API, we can filter out anything you don't need to read, so a users time is saved and a discussion from the past is easier to recall.

**Diversity** \
Today, humans manually transcribe audio/video content on platforms like Discord, Instagram, and Reddit. These transcriptions are quite useful for those who are deaf or hard of hearing. People from countries with limited access to the internet are known to use transcriptions to consume media as the poor connection speeds are unable to load the video content. The functionality our bot provides can improve the lives of these people. A human is not always there to transcribe media, but an AI can always be there.

## Challenges we ran into

Getting the individual pieces of the program working took much less time compared to figuring out how to get them to communicate. We tried many libraries designed to allow JavaScript to talk with Python, but none worked perfectly. We ended up creating our own solution using the functionality built into JavaScript.
We also extensively tested the NLP text summarization models, as there were a lot of parameters in the model we had to tune to be able to get summaries that fit our use case.

## Accomplishments that we're proud of

We are proud that we ended up with a product, in roughly 24 hours, that works exactly as we intended it to, right from an idea and converting it into a usable tool with plenty of utilities waiting to be added in the near future. Despite the numerous errors we faced, we never gave up and ultimately were genuinely satisfied with all of our efforts as a team.

## What we learned

Hack the Valley being the first-ever hackathon that we have all participated in has taught us a lot, whether it be implementing new APIs and ML models for the first time or just handling failures. We went into this challenge doubtful that we would be able to write even a half-functioning program, but we surprised ourselves with how much we can get done if we put in serious effort. We learned to stay together and solve problems that we have never encountered in the past and our genuinely proud of the hard work all of us put in.

## What's next for Discribe

Our main goal is to expand the use of our bot to other social media platforms, mainly Reddit. With some code refactoring, the bot can start transcribing videos that are posted automatically. We could achieve this scale using hosting services such as Microsoft Azure. To extend its utilities, we will be working to detect and extract useful and critical information such as key topics, phone numbers, addresses, or maybe we could even separate voices in the transcription, to give good context about who is speaking.
## Inspiration

*For a more expressive texting experience.*
Texting often lacks the non-verbal cues present in talking face-to-face, especially when using Speech-to-Text programs. Emojis help show intent, and clarify ambiguous texts; they are the non-verbal cues of texting. Naturally, some Speech-to-Text programs do support emojis, but they rely on phrases like "happy face" or "sad face." This could become problematic/unnatural/inconvenient while speaking. We were inspired to make an Discord bot that streamlines Speech-to-Text and emojis by analyzing real-time facial expressions and converting them into emojis, while also transcribing voice to text.

It also helps people who have a hard time picking up social cues. This one is for you, Alex 😉.

## What it does

Our program takes a video file and transcribes it with AssemblyAI API. Then, using the MoviePy library, we sliced up the video file into frames and into images. Next, we use Google's Cloud Vision API to detect faces in these images, as well as their emotions. Finally, we return the text back to the user along with the emoji.

## How we built it

We built with Python, AssemblyAI, and Cloud Vision.

## Challenges we ran into

Originally, we planned on making a discord bot, but we did not manage to captured both voice and video. Thus, we decided to put the Discord integration aside, and focus on the backend of the application.

## Accomplishments that we're proud of

We changed the course of our project with a few hours left in the hackathon, and still managed to get a few hours of sleep.

## What we learned

* How to create webhooks
* How to make discord bots
* How to make API calls and deal with JSON responses

## What's next for Teemo

* To incorporate the ability to analyze the tone of the transcript.
* Eventual integration with Discord (or some other frontend/app)
* More expressive emojis (maybe even non-emotion emojis)
## **Inspiration:**

Our inspiration stemmed from the realization that the pinnacle of innovation occurs at the intersection of deep curiosity and an expansive space to explore one's imagination. Recognizing the barriers faced by learners—particularly their inability to gain real-time, personalized, and contextualized support—we envisioned a solution that would empower anyone, anywhere to seamlessly pursue their inherent curiosity and desire to learn.

## **What it does:**

Our platform is a revolutionary step forward in the realm of AI-assisted learning. It integrates advanced AI technologies with intuitive human-computer interactions to enhance the context a generative AI model can work within. By analyzing screen content—be it text, graphics, or diagrams—and amalgamating it with the user's audio explanation, our platform grasps a nuanced understanding of the user's specific pain points. Imagine a learner pointing at a perplexing diagram while voicing out their doubts; our system swiftly responds by offering immediate clarifications, both verbally and with on-screen annotations.

## **How we built it**:

We architected a Flask-based backend, creating RESTful APIs to seamlessly interface with user input and machine learning models. Integration of Google's Speech-to-Text enabled the transcription of users' learning preferences, and the incorporation of the Mathpix API facilitated image content extraction. Harnessing the prowess of the GPT-4 model, we've been able to produce contextually rich textual and audio feedback based on captured screen content and stored user data. For frontend fluidity, audio responses were encoded into base64 format, ensuring efficient playback without unnecessary re-renders.

## **Challenges we ran into**:

Scaling the model to accommodate diverse learning scenarios, especially in the broad fields of maths and chemistry, was a notable challenge. Ensuring the accuracy of content extraction and effectively translating that into meaningful AI feedback required meticulous fine-tuning.

## **Accomplishments that we're proud of**:

Successfully building a digital platform that not only deciphers image and audio content but also produces high utility, real-time feedback stands out as a paramount achievement. This platform has the potential to revolutionize how learners interact with digital content, breaking down barriers of confusion in real-time. One of the aspects of our implementation that separates us from other approaches is that we allow the user to perform ICL (In Context Learning), a feature that not many large language models don't allow the user to do seamlessly.

## **What we learned**:

We learned the immense value of integrating multiple AI technologies for a holistic user experience. The project also reinforced the importance of continuous feedback loops in learning and the transformative potential of merging generative AI models with real-time user input.