## Inspiration

Have you ever wished you had…another you?

This thought has crossed all of our heads countless times as we find ourselves swamped in too many tasks, unable to keep up in meetings as information flies over our head, or wishing we had the feedback of a third perspective.

Our goal was to build an **autonomous agent** that could be that person for you — an AI that learns from your interactions and proactively takes **actions**, provides **answers**, offers advice, and more, to give back your time to you.

## What it does

Ephemeral is an **autonomous AI agent** that interacts with the world primarily through the modality of **voice**. It can sit in on meetings, calls, anywhere you have your computer out.

It’s power is the ability to take what it hears and proactively carry out repetitive actions for you such as be a real-time AI assistant in meetings, draft emails directly in your Google inbox, schedule calendar events and invite attendees, search knowledge corpuses or the web for answers to questions, image generation, and more.

Multiple users (in multiple languages!) can use the technology simultaneously through the server/client architecture that efficiently handles multiprocessing.

## How we built it

![link](https://i.imgur.com/PatcdIi.png)

**Languages**: Python ∙ JavaScript ∙ HTML ∙ CSS

**Frameworks and Tools**: React.js ∙ PyTorch ∙ Flask ∙ LangChain ∙ OpenAI ∙ TogetherAI ∙ Many More

### 1. Audio to Text

We utilized OpenAI’s Whisper model and the python speech\_recognition library to convert audio in real-time to text that can be used by downstream functions.

### 2. Client → Server via Socket Connection

We use socket connections between the client and server to pass over the textual query to the server for it to determine a particular action and action parameters. The socket connections enable us to support multiprocessing as multiple clients can connect to the server simultaneously while performing concurrent logic (such as real-time, personalized agentic actions during a meeting).

### 3. Neural Network Action Classifier

We trained a neural network from scratch to handle the multi-class classification problem that is going from text to action (or none at all). Because the agent is constantly listening, we need a way to efficiently and accurately determine if each transcribed chunk necessitates a particular action (if so, which?) or none at all (most commonly).

We generated data for this task utilizing data augmentation sources such as ChatGPT (web).

### 4. LLM Logic: Query → Function Parameters

We use in-context learning via few-shot prompting and RAG to query the LLM for various agentic tasks. We built a RAG pipeline over the conversation history and past related, relevant meetings for context. The agentic tasks take in function parameters, which are generated by the LLM.

### 5. Server → Client Parameters via Socket Connection

We pass back the function parameters as a JSON object from the server socket to the client.

### 6. Client Side Handler: API Call

A client side handler receives a JSON object that includes which action (if any) was chosen by the Action Planner in step 3, then passes control to the appropriate handler function which handles authorizations and makes API calls to various services such as Google’s Gmail Client, Calendar API, text-to-speech, and more.

### 7. Client Action Notifications → File (monitored by Flask REST API)

After the completion of each action, the client server writes the results of the action down to a file which is then read by the React Web App to display ephemeral updates on a UI, in addition to suggestions/answers/discussion questions/advice on a polling basis.

### 8. React Web App and Ephemeral UI

To communicate updates to the user (specifically notifications and suggestions from Ephemeral), we poll the Flask API for any updates and serve it to the user via a React web app. Our app is called Ephemeral because we show information minimally yet expressively to the user, in order to promote focus in meetings.

## Challenges we ran into

We spent a significant amount of our time optimizing for lower latency, which is important for a real-time consumer-facing application. In order to do this, we created sockets to enable 2-way communication between the client(s) and the server. Then, in order to support concurrent and parallel execution, we added support for multithreading on the server-side.
Choosing action spaces that can be precisely articulated enough in text such that a language model can carry out actions was a troublesome task. We went through a lot of experimentation on different tasks to figure out which would have the highest value to humans and also the highest correctness guarantee.

## Accomplishments that we're proud of

Successful integration of numerous OSS and closed source models into a working product, including Llama-70B-Chat, Mistral-7B, Stable Diffusion 2.1, OpenAI TTS, OpenAI Whisper, and more.
Integration of real actions that we can see ourselves directly using was very cool to see go from a hypothetical to a reality. The potential for impact of this general workflow in various domains is not lost on us, as while the general productivity purpose stands, there are many more specific gains to be seen in fields such as digital education, telemedicine, and more!

## What we learned

The possibility of powerful autonomous agents to supplement human workflows signals the shift of a new paradigm where more and more our imprecise language can be taken by these programs and turned into real actions on behalf of us.

## What's next for Ephemeral

An agent is only constrained by the size of the action space you give it. We think that Ephemeral has the potential to grow boundlessly as more powerful actions are integrated into its planning capabilities and it returns more of a user’s time to them.
## Inspiration

Many people feel unconfident, shy, and/or awkward doing interview speaking. It can be challenging for them to know how to improve and what aspects are key to better performance. With Talkology, they will be able to practice in a rather private setting while receiving relatively objective speaking feedback based on numerical analysis instead of individual opinions. We hope this helps more students and general job seekers become more confident and comfortable, crack their behavioral interviews, and land that dream offer!

## What it does

* Gives users interview questions (behavioural, future expansion to questions specific to the job/industry)
* Performs quantitative analysis of users’ responses using speech-to-text & linguistic software package praat to study acoustic features of their speech
* Displays performance metrics with suggestions in a user-friendly, interactive dashboard

## How we built it

* React/JavaScript for the frontend dashboard and Flask/Python for backend server and requests
* My-voice-analysis package for voice analysis in Python
* AssemblyAI APIs for speech-to-text and sentiment analysis
* MediaStream Recording API to get user’s voice recordings
* Figma for the interactive display and prototyping

## Challenges we ran into

We went through many conversations to reach this idea and as a result, only started hacking around 8AM on Saturday. On top of this time constraint layer, we also lacked experience in frontend and full stack development. Many of us had to spend a lot of our time debugging with package setup, server errors, and for some of us even M1-chip specific problems.

## Accomplishments that we're proud of

This was Aidan’s first full-stack application ever. Though we started developing kind of late in the event, we were able to pull most of the pieces together within a day of time on Saturday. We really believe that this product (and/or future versions of it) will help other people with not only their job search process but also daily communication as well. The friendships we made along the way is also definitely something we cherish and feel grateful about <3

## What we learned

* Aidan: Basics of React and Flask
* Spark: Introduction to Git and full-stack development with sprinkles of life advice
* Cathleen: Deeper dive into Flask and React and structural induction
* Helen: Better understanding of API calls & language models and managing many different parts of a product at once

## What's next for Talkology

We hope to integrate computer vision approaches by collecting video recordings (rather than just audio) to perform analysis on hand gestures, overall posture, and body language. We also want to extend our language analysis to explore novel models aimed at performing tone analysis on live speech. Apart from our analysis methods, we hope to improve our question bank to be more than just behavioural questions and better cater to each user's specific job demands. Lastly, there are general loose ends that could be easily tied up to make the project more cohesive, such as integrating the live voice recording functionality and optimizing some remaining components of the interactive dashboard.
## Inspiration

As members of immigrant families, we often encounter the significant challenge of language barriers when communicating with family members in our home countries. This issue has become more prevalent since the onset of the COVID-19 pandemic, leading to missed opportunities and lack of meaningful connections with long distance family members. While the straightforward solution might seem to be speaking the language at home and utilizing educational platforms like DuoLingo or Babbel for practice, it's not as simple as it appears. People often lose motivation or focus solely on obtaining the correct answers on these apps. This challenge inspired us to create a more engaging and interactive conversational companion. By offering users the opportunity to engage in real conversations, this tool enhances language practice and improves proficiency through repeated speaking iterations.

## What it does

Our web platform offers users the opportunity to learn various languages through a tailored lesson plan upon selecting a language. It includes sub-lessons that involve engagement with articles or short videos on specific topics. This design emulates the interactive experience of a traditional classroom setting.

Following the consumption of these materials, users engage with AI Chat Companion for a unique twist on language learning: voice-to-voice conversations. Users speak into the microphone, and their input is processed in real time. The AI companion analyzes the spoken responses and replies with audio feedback. Conversations are seamlessly conducted and can be concluded using a language-specific keyword(eg "terminador" for Spanish).

The AI focuses primarily on understanding, with grammar and punctuation as secondary feedback priorities. We had the intent of fostering a conversational environment that enhances speaking skills and provides a comprehensive speaking experience for learners.

## How we built it

Our product was developed using a combination of various technologies and frameworks to ensure an ideal interactive user experience.

For image conversion, we utilized Covertio, which is a tool used for converting JPEG images into SVG format, which allows us to integrate a high quality vector graphic into the website's UI. The user interface was designed using Reflex, which is a flexible, open-source, full stack python framework which allowed for an efficient deployment of the app. The framework unified front and backend, which allowed us to implement both purely in python. We adopted a sidebar template from Reflex to organize content and ensure an intuitive navigation experience for users.

Python was the core of our AI functionality, where the chatbot service and speech-to-text and text-to-speech processing was developed using Python, Open AI APIs, and Speech Text from Google Cloud. The use of Python allowed for access to diverse libraries and frameworks that support AI. The chatbot was integrated using Python enabling real time interactions with users. The component was crucial for simulating the conversation experience.

For speech recognition and synthesis, we used Google Cloud's Speech Text APIs, which provided the backbone for our backend implementation for accurate NLP training, and our basis for prompt engineering. The speech detection feature on google cloud allowed for strong capabilities in audio processing. The Open AI API allowed for real time translation within a language during conversations.

## Challenges we ran into

The development process involved navigating through a complex landscape of technical challenges. One of our initial struggles was dealing with framework compatibility. The integration of Reflex, which is a significantly different framework compared to Node.js and React, led to a fairly substantial learning curve. Understanding the framework required time to understand the components and best practices to ensure a responsive design. We were able to perform comprehensive research as a team and speak with the sponsors for any issues we had. We performed extensive testing to ensure compatibility.

Configuring speech-to-text service to accurately recognize and process spoken language presented difficulties with dialects, accents, and other background noise. Effective prompt engineering that elicited useful responses form the chatbot while ensuring flow of conversation was difficult. We were able to utilize Google Cloud's Speech Text API for its real time speech recognition capabilities. We applied practices in natural language processing and conducted user testing to refine interactions.

Deciding on the best platform for speech-to-text development involved in evaluating various options based on latency was another difficulty, along with ensuring the chat bot remained on topic with coherent responses were some of the final backend challenges.

## Accomplishments that we're proud of

We are proud of figuring out and programming an organized UI platform using an architecture and framework we had no experience with. Working together as a team and problem solving with each through issues in speech to text interpretation, prompt engineering, and training the LLM is something we feel great about.

## What we learned

We learned that pivoting to a brand new UI platform with no prior knowledge about is a challenging task, especially regarding the difficulty of having a comprehensive understanding within the duration of the hackathon. We learned how to implement OpenAI APIs and Speech Text APIs from Google Cloud, and the development behind creating speech(from user) to speech(chatbot) communication. We also learned how to divide and conquer, as well as work together on tasks that were difficult.

## What's next for Polyglot.AI

Fine tuning the LLM for a more structured response from the chat bot would make the learning experience more ideal. Improving our UIUX process and design it to be more interactive and flow more effectively. Improving the environment of the platform to make the communication purely as speech to speech.