**Elevate your foreign language fluency with tailored guidance and individualized verbal conversation practice on Ekko.**

## Inspiration

Multilingualism is in. In today’s interconnected world, the ability to communicate in multiple languages is not only valuable, but imperative to personal and professional success. Whether it be conversing with business professionals during international commerce exchanges, preparing for standardized school language exams, serving as a journalist or diplomat on the international stage, or simply wanting to converse more fluently with your loved ones, everybody can benefit from conversing in a foreign language. However, traditional curriculum-based language learning methods rely on repetitive exercises and lack personalization, overlooking verbal fluency markers such as verbal precision, language variation, and personal goals. A key indication of an advanced speaker is the ability to engage in debates and convey subtle shades of meaning effectively, a skill that cannot be developed solely through the use of apps centered around memorization.

As a 2nd generation Mandarin and Cantonese speaker, one of our team members realized firsthand how difficult it was to maintain fluency of foreign languages in university. Additionally, while visiting her grandmother in the ICU at a reputable San Francisco hospital, another team member noticed that it was frustrating—and potentially life threatening—for non-English speaking patients to communicate their needs to care providers since these care providers are not trained conversationally since they received very rudimentary academic foreign language training. Although these bilingual care providers were technically licensed to care for non-English speaking patients, most of them who learned the foreign language as a second language were unable to demonstrate spoken proficiency and cultural awareness outside of a classroom context. These experiences all led us to develop Ekko.

Whether you’re a busy professional looking to enhance your global marketability or a student aiming to broaden your cultural horizons during study abroad, Ekko enables you to access verbal language practice anytime, anywhere, offering you the personalization and flexibility to learn at your own pace and develop as a global citizen.

## What it does

Introducing **Ekko**: a personalized real time AI vocal chatbot that assesses your vocal language fluency.

With Ekko, you only talk about what you actually want to talk about. Once you enter your basic onboarding information into Ekko such as your learning goals and interests, the app will then prompt you to a simple user interface where you can start your conversation. After each response, Ekko will then give personalized feedback on your conversational performance by catching your errors and providing you with an ACTFL based proficiency level. Conversations and feedback are personalized to your learning goals; for example, if you are using Ekko to prepare for career oriented work purposes, Ekko would generate prompts that you’d likely encounter in the workplace and the feedback would likely be centered around making your diction more formal. Similarly, if you were simply using Ekko to converse with friends and family, conversation topics and corrections provided by the chatbot would be more casual.

Ekko saves your speaking errors and transfers those language specific content errors to tailor feedback to your language learning goals. For example, if you were to say *“me llamo es Cole”* as opposed to the correct version: *“me llamo Cole”*, Ekko would save that error to check in the future. Using this unique feature, Ekko also makes connections between your learning language of interest and languages you currently speak (inputted during the onboarding process), drawing parallels between the two.

Similarly, if you were to consecutively respond with singular word responses, Ekko would suggest that you vary your sentence structure to maximize the effectiveness of the conversation.

Unlike pre-existing language learning applications such as Duolingo, Ekko is *not* based on a curriculum, meaning that you take full reign of the conversation and practice.

## How we built it

To make Ekko as capable as possible, we used a combination of many AI and machine learning technologies—most of which we had never used before.

Because Ekko’s main value proposition is its conversational aspect, it was important that conversing with the platform is as natural as possible. This included using a state-of-the-art text-to-speech model, powered by ElevenLabs, as well as speech-to-text, powered by Deepgram. The combination of these two technologies made natural conversation on Ekko a seamless experience.

Processing speed was also of utmost importance to us to make the conversations feel natural. Hence, the obvious choice for us was to power our backend using Bun. Specifically, we’re running an Elysia.js server to interface with our ML and large language models for incredibly fast performance. This strategic choice contributed to Ekko's impressive performance and responsiveness during interactions.

Regarding large language models, Ekko chose to go full open-source thanks to Together.AI. We’re using the "NousResearch/Nous-Hermes-2-Yi-34B" model to generate responses from the AI agent, as well as "togethercomputer/m2-bert-80M-32k-retrieval" for text embeddings. These models were blazingly fast and out-performed the multitude of other models we tested for these purposes.

To store the data we collected, we chose to use the Convex.dev platform. We’re leveraging their database and authentication services, as well as function calling and vector database. Using Convex enabled us to build a complex platform with many simultaneous and interconnected processes in such a limited time span.

In order to classify the user’s proficiency, we built a text classification model using scikit learn. To train this model, we generated a synthetic dataset of hypothetical conversations that corresponded to specific ACTFL Proficiency guidelines using Together.AI’s "NousResearch/Nous-Hermes-2-Yi-34B" model. This model, hosted on GCP Vertex AI platform, enables us to specifically denote the user’s progress as they reach fluency.

Altogether, Ekko's development is characterized by a comprehensive integration of state-of-the-art technologies. The emphasis on natural conversation, swift processing, open-source language models, efficient data handling through Convex.dev, and a proficiency-classifying text model collectively contribute to Ekko's prowess as an advanced conversational language learning platform powered by frontier tech.

## Business model

In regards to our business model, we initially looked into adopting a Freemium model, but ultimately steered away from that inclination due to not wanting to exacerbate accessibility issues in the edtech space. For now, we intend for all Ekko features to be free of charge, and eventually rely on community partnerships and sponsorships with relatively small organizations such as out patient clinics in order to cover costs. In the future during the reiteration phase, we also plan on hosting a donation platform to raise money for our developing team, as well as to purchase technology in underprivileged schools so that students worldwide can use Ekko. We also want to look into partnerships with larger organizations that would benefit from improved language fluency services such as hotels and universities.

## Challenges we ran into

One of the major challenges we encountered was finding an adequate fluency metric to score user responses. While percentages and other numerical metrics seemed like an obvious choice, this would also mean that the longer a user were to maintain a conversation (typically holding a longer conversation is a good thing when practicing foreign languages), the higher percent error they’d receive, thus deterring users from talking for longer periods of time. We eventually settled on the idea of using qualitative feedback based on the well-established ACTFL Language Speaking category rankings that contained specific comprehension and fluency requirements under each conversation difficulty level. The scoring would be based off the average ACTFL score of the five most recent responses provided.

Our team also raised larger scale questions pertaining to stuttering and speech impediments, as many fluent speakers often naturally stutter while talking and the STT model could interpret that as lack of fluency. Moreover, usage of slang is also something that we need to look into a bit further, as the current system struggles to interpret colloquial vernacular.

## What we learned

Through developing Ekko, we learned that different languages present different challenges with TTS that we must reconsider when building past our MVP.

We also learned the major advantages that active conversation has over repetitive exercises when practicing a foreign language, as active conversation provides learners the opportunity to contextual learning in practical and authentic situations through immediate correction while repetitive exercises solely focus on reinforcing specific patterns and foundational drills and lack spontaneity of real-life language.

## What's next for Ekko

During our next iteration of Ekko, we hope to also launch a real-time typing version of our personalized chatbot that would simulate your ideal interlocutor in both content and formality. We are also looking to implement a feature that encourages users to utilize figurative language in their speech. For example, if the user were using Ekko to improve their English proficiency and they told our chatbot "*It’s raining very hard outside,*” Ekko would highlight that sentence and perhaps suggest: “*It’s raining cats and dogs*” or the more casual “*It’s pouring.*" Another feature we are looking to implement post-MVP is a time suggestion feature, as it is a useful skill to show a comprehensive understanding of the other person’s input while also keeping responses pertinent and cutting off unnecessary fluff. This feature would especially come in handy to those preparing for professional interviews.

Now more on Ekko’s social component. Our team would like to develop a social component to Ekko integrating a gamified social element that allows students to build profiles, connect with other students, and compare streaks with friends. We would also consider gamifying the conversations with fun interaction challenge modes simulating Heads Up or Hot Seat. This would not only incentivize users to practice their verbal communication even more, but would also be especially helpful to those using Ekko to prepare for less formal conversational settings.

In regards to getting the word about Ekko out there, our team would launch a guerilla marketing campaign, pushing out content on all social media platforms and attending in-person hackathons and conventions to get initial feedback during beta testing.

In addition to partnerships with small clinics and healthcare providers, we would also gradually partner with more outside organizations such as university residential housing and career services, refugee councils helping young asylum seekers, and larger international corporations to implement Ekko into their daily regimens.

Lastly, we would also like to launch a donation platform to support the developing team and generate donations for tech for underprivileged schools so they can continue using Ekko.

## Ethical Discussion

First and foremost, the responsible use of large language models (LLMs) has engendered immense ethical debate, as they can inadvertently perpetuate representation biases from the data used to train them, thereby amplifying existing linguistic and cultural prejudice. Thus, it is imperative that developers—especially developers of language learning applications—stress the importance of cultural sensitivity, empathy, and respect for linguistic diversity in language learning communities. LLMs also pose security risks that must be mitigated through robust cybersecurity measures, and as with any web application, concerns regarding data privacy and security raise concerns about the safeguarding of personal user information within educational platforms. However, we see Ekko being a safer alternative to similar platforms such as TalkAbroad, as users are chatting verbally with our chatbot rather than on live video call with an individual they are unfamiliar with.

Additionally, another key factor in promoting inclusivity in educational technology is ensuring accessibility to technologies and devices. Since we intend on Ekko being used in underserved school communities where access to devices is not always guaranteed, in the future, we want to collect donations and sponsors to purchase devices for underprivileged schools so they can continue using Ekko.

Furthermore, current speech-to-text platforms overlook individuals impacted by speech impediments, creating potential barriers to participation. Through rounds of reiteration and beta testing, we hope to eventually develop a version of Ekko that accounts for individuals with speech and learning disabilities.

Additionally, the development of Ekko could financially impact those who rely on virtual conversation exchange provider services such as TalkAbroad for supplementary income.

Lastly, monetization strategies and pricing models such as the popular Freemium model can exacerbate educational inequities and access. Though our team has collectively decided to offer all our services free of charge, that does put us at a stalemate when discussing how we will monetize. Although the potential positive impact of Ekko is rife, it is still crucial that we are diligent in navigating these complexities. It is imperative for us to address these issues conscientiously, ensuring that educational language fluency technologies remain accessible, equitable, and respectful of diverse linguistic and cultural backgrounds.

## Citations

<https://arxiv.org/pdf/2307.06435.pdf>
## Inspiration

We were motivated to tackle linguistic challenges in the educational sector after juxtaposing our personal experience with current news.

There are currently over 70 million asylum seekers, refugees, or internally-displaced people around the globe, and this statistic highlights the problem of individuals from different linguistic backgrounds being forced to assimilate into a culture and language different than theirs. As one of our teammates was an individual seeking a new home in a new country, we had first hand perspective at how difficult this transition was. In addition, our other team members had volunteered extensively within the educational system in developing communities, both locally and globally, and saw a similar need with individuals being unable to meet the community’s linguistics standards.

We also iterated upon our idea to ensure that we are holistically supporting our communities by making sure we consider the financial implications of taking the time to refine your language skills instead of working.

## What it does

Fluently’s main purpose is to provide equitable education worldwide. By providing a user customized curriculum and linguistic practice, students can further develop their understanding of their language. It can help students focus on areas where they need the most improvement. This can help them make progress at their own pace and feel more confident in their language skills while also practicing comprehension skills. By using artificial intelligence to analyze pronunciation, our site provides feedback that is both personalized and objective.

## How we built it

Developing the web application was no easy feat.

As we were searching for an AI model to help us through our journey we stumbled upon OpenAI, specifically Microsoft Azure’s cognitive systems that utilize OpenAI’s comprehensive abilities in language processing. This API gave us the ability to analyze voice patterns and fluency and transcribe passages that are mentioned in the application. Figuring out the documentation as well as how the AI will be interacting with the user was most important for us to execute properly since the AI would be acting as the tutor/mentor for the students in these cases. We developed a diagram that would break down the passages read to the student phonetically and give them a score of 100 for how well each word was pronounced based on the API’s internal grading system. As it is our first iteration of the web app, we wanted to explore how much information we could extract from the user to see what is most valuable to display to them in the future.

Integrating the API with the web host was a new feat for us as a young team. We were confident in our python abilities to host the AI services and found a library by the name of Flask that would help us write html and javascript code to help support the front end of the application through python. By using Flask, we were able to host our AI services with python while also continuously managing our front end through python scripts.

This gave room for the development of our backend systems which are Convex and Auth0. Auth0 was utilized to give members coming into the application a unique experience by having them sign into a personalized account. The account is then sent into the Convex database to be used as a storage base for their progress in learning and their development of skills over time. All in all, each component of the application from the AI learning models, generating custom passages for the user, to the backend that communicated between the Javascript and Python server host that streamlines the process of storing user data, came with its own challenges but came together seamlessly as we guide the user from our simple login system to the passage generator and speech analyzer to give the audience constructive feedback on their fluency and pronunciation.

## Challenges we ran into

As a majority beginning team, this was our first time working with many of the different technologies, especially with AI APIs. We need to be patient working with key codes and going through an experiment process of trying different mini tests out to then head to the major goal that we were headed towards. One major issue that we faced was the visualization of data to the user. We found it hard to synthesize the analysis that was done by the AI to translate to the user to make sure they are confident in what they need to improve on. To solve this problem we first sought out how much information we could extract from the AI and then in future iterations we would simply display the output of feedback.

Another issue we ran into was the application of convex into the application. The major difficulty came from developing javascript functions that would communicate back to the python server hosting the site. This was resolved thankfully; we are grateful for the Convex mentors at the conference that helped us develop personalized javascript functions that work seamlessly with our Auth0 authentication and the rest of the application to record users that come and go.

## Accomplishments that we're proud of:

One accomplishment that we are proud of was the implementation of Convex and Auth0 with Flask and Python. As python is a rare language to host web servers in and isn't the primary target language for either service, we managed to piece together a way to fit both services into our project by collaboration with the team at Convex to help us out. This gave way to a strong authentication platform for our web application and for helping us start a database to store user data onto.

Another accomplishment was the transition of using a React Native application to using Flask with Python. As none of the group has seen Flask before or worked for it for that matter, we really had to hone in our abilities to learn on the fly and apply what we knew prior about python to make the web app work with this system.

Additionally, we take pride in our work with OpenAI, specifically Azure. We researched our roadblocks in finding a voice recognition AI to implement our natural language processing vision. We are proud of how we were able to display resilience and conviction to our overall mission for education to use new technology to build a better tool.

## What we learned

As beginners at our first hackathon, not only did we learn about the technical side of building a project, we were also able to hone our teamwork skills as we dove headfirst into a project with individuals we had never worked with before.

As a group, we collectively learned about every aspect of coding a project, from refining our terminal skills to working with unique technology like Microsoft Azure Cognitive Services. We also were able to better our skillset with new cutting edge technologies like Convex and OpenAI.

We were able to come out of this experience not only growing as programmers but also as individuals who are confident they can take on the real world challenges of today to build a better tomorrow.

## What's next?

We hope to continue to build out the natural language processing applications to offer the technology in other languages. In addition, we hope to hone to integrate other educational resources, such as videos or quizzes to continue to build other linguistic and reading skill sets. We would also love to explore the cross section with gaming and natural language processing to see if we can make it a more engaging experience for the user. In addition, we hope to expand the ethical considerations by building a donation platform that allows users to donate money to the developing community and pay forward the generosity to ensure that others are able to benefit from refining their linguistic abilities. The money would then go to a prominent community in need that uses our platform to fund further educational resources in their community.

## Bibliography

United Nations High Commissioner for Refugees. “Global Forced Displacement Tops 70 Million.” UNHCR, UNHCR, The UN Refugee Agency, <https://www.unhcr.org/en-us/news/stories/2019/6/5d08b6614/global-forced-displacement-tops-70-million.html>.
## Inspiration

We take our inspiration from our everyday lives. As avid travellers, we often run into places with foreign languages and need help with translations. As avid learners, we're always eager to add more words to our bank of knowledge. As children of immigrant parents, we know how difficult it is to grasp a new language and how comforting it is to hear the voice in your native tongue. LingoVision was born with these inspirations and these inspirations were born from our experiences.

## What it does

LingoVision uses AdHawk MindLink's eye-tracking glasses to capture foreign words or sentences as pictures when given a signal (double blink). Those sentences are played back in an audio translation (either using an earpiece, or out loud with a speaker) in your preferred language of choice. Additionally, LingoVision stores all of the old photos and translations for future review and study.

## How we built it

We used the AdHawk MindLink eye-tracking classes to map the user's point of view, and detect where exactly in that space they're focusing on. From there, we used Google's Cloud Vision API to perform OCR and construct bounding boxes around text. We developed a custom algorithm to infer what text the user is most likely looking at, based on the vector projected from the glasses, and the available bounding boxes from CV analysis.

After that, we pipe the text output into the DeepL translator API to a language of the users choice. Finally, the output is sent to Google's text to speech service to be delivered to the user.

We use Firebase Cloud Firestore to keep track of global settings, such as output language, and also a log of translation events for future reference.

## Challenges we ran into

* Getting the eye-tracker to be properly calibrated (it was always a bit off than our view)
* Using a Mac, when the officially supported platforms are Windows and Linux (yay virtualization!)

## Accomplishments that we're proud of

* Hearing the first audio playback of a translation was exciting
* Seeing the system work completely hands free while walking around the event venue was super cool!

## What we learned

* we learned about how to work within the limitations of the eye tracker

## What's next for LingoVision

One of the next steps in our plan for LingoVision is to develop a dictionary for individual words. Since we're all about encouraging learning, we want to our users to see definitions of individual words and add them in a dictionary.

Another goal is to eliminate the need to be tethered to a computer. Computers are the currently used due to ease of development and software constraints. If a user is able to simply use eye tracking glasses with their cell phone, usability would improve significantly.