Our team came together out of a shared love for language learning and shared dissatisfaction with how modern applications are handling it. The market is oversaturated with beginner apps that repeat common words and vocabulary until it sticks that day, and you probably forget it the next. Seldom do they focus on what truly differentiates a novice from an expert: understanding tone, situational context, regional dialects, accents, and cultural sayings. But rather than attempting to provide those resources as a group of 4 people who aren't very fluent in many languages, why not allow *actual* natives to teach what they know best? And better yet, why not allow them to be taught by other speakers looking to improve in the language you have proficiency in?

*LanEx* is a platform intended to facilitate language exchange through a pairing system and guided chatrooms using context and tone-based prompts. It attempts to address the deficiencies we observe within the language learning market and decentralize teaching, allowing intermediate learners to grab from multiple native sources when attaining competency. The pairing system is based on a set of questions we ask upon profile creation, such as what languages they are proficient in and to what level, what languages they want to learn, preferences for partners, etc.

One of the special features of our application is that it incorporates the best qualities of a tutoring system without its pitfalls. The interactions between users on this website are mutually beneficial. For instance, if a person knows English and wants to learn French, they would be paired with a partner that speaks French and wants to learn English. Through the pairing, both parties can benefit from learning while substituting payments with providing intrinsically-held knowledge; a deal which makes language learning altogether more inclusive and accessible.

For the front end, we utilized HTML, CSS, Javascript, and the Bootstrap framework, as well as Canva for logos and designs. As for the backend, we used SQL, Python, and the framework Flask, although our demo site is largely hard-coded due to time constraints in learning Flask. To edit and collaborate, we used Visual Studio Code and GitHub—both proving to be immensely helpful.

By far the most challenging aspect was learning Flask and other frameworks that would build on our prior knowledge of HTML, CSS, Javascript, and Python. While we understood how to implement it on a basic level through the help of mentors, workshops, and tutorials, it was difficult learning how to execute most of our logic using this framework. Despite this, the fact that we implemented our designs into a working application is an accomplishment on its own, and it goes to show how much we've grown in technical ability and creativity this weekend. While we may not be Flask or SQL experts, we are far more knowledgeable than before in those technologies. Going forward, we hope that LanEx can be expanded to reach a global audience of multiple different languages and speakers, but before that, even an audience of three would be amazing.
## Inspiration

As members of immigrant families, we often encounter the significant challenge of language barriers when communicating with family members in our home countries. This issue has become more prevalent since the onset of the COVID-19 pandemic, leading to missed opportunities and lack of meaningful connections with long distance family members. While the straightforward solution might seem to be speaking the language at home and utilizing educational platforms like DuoLingo or Babbel for practice, it's not as simple as it appears. People often lose motivation or focus solely on obtaining the correct answers on these apps. This challenge inspired us to create a more engaging and interactive conversational companion. By offering users the opportunity to engage in real conversations, this tool enhances language practice and improves proficiency through repeated speaking iterations.

## What it does

Our web platform offers users the opportunity to learn various languages through a tailored lesson plan upon selecting a language. It includes sub-lessons that involve engagement with articles or short videos on specific topics. This design emulates the interactive experience of a traditional classroom setting.

Following the consumption of these materials, users engage with AI Chat Companion for a unique twist on language learning: voice-to-voice conversations. Users speak into the microphone, and their input is processed in real time. The AI companion analyzes the spoken responses and replies with audio feedback. Conversations are seamlessly conducted and can be concluded using a language-specific keyword(eg "terminador" for Spanish).

The AI focuses primarily on understanding, with grammar and punctuation as secondary feedback priorities. We had the intent of fostering a conversational environment that enhances speaking skills and provides a comprehensive speaking experience for learners.

## How we built it

Our product was developed using a combination of various technologies and frameworks to ensure an ideal interactive user experience.

For image conversion, we utilized Covertio, which is a tool used for converting JPEG images into SVG format, which allows us to integrate a high quality vector graphic into the website's UI. The user interface was designed using Reflex, which is a flexible, open-source, full stack python framework which allowed for an efficient deployment of the app. The framework unified front and backend, which allowed us to implement both purely in python. We adopted a sidebar template from Reflex to organize content and ensure an intuitive navigation experience for users.

Python was the core of our AI functionality, where the chatbot service and speech-to-text and text-to-speech processing was developed using Python, Open AI APIs, and Speech Text from Google Cloud. The use of Python allowed for access to diverse libraries and frameworks that support AI. The chatbot was integrated using Python enabling real time interactions with users. The component was crucial for simulating the conversation experience.

For speech recognition and synthesis, we used Google Cloud's Speech Text APIs, which provided the backbone for our backend implementation for accurate NLP training, and our basis for prompt engineering. The speech detection feature on google cloud allowed for strong capabilities in audio processing. The Open AI API allowed for real time translation within a language during conversations.

## Challenges we ran into

The development process involved navigating through a complex landscape of technical challenges. One of our initial struggles was dealing with framework compatibility. The integration of Reflex, which is a significantly different framework compared to Node.js and React, led to a fairly substantial learning curve. Understanding the framework required time to understand the components and best practices to ensure a responsive design. We were able to perform comprehensive research as a team and speak with the sponsors for any issues we had. We performed extensive testing to ensure compatibility.

Configuring speech-to-text service to accurately recognize and process spoken language presented difficulties with dialects, accents, and other background noise. Effective prompt engineering that elicited useful responses form the chatbot while ensuring flow of conversation was difficult. We were able to utilize Google Cloud's Speech Text API for its real time speech recognition capabilities. We applied practices in natural language processing and conducted user testing to refine interactions.

Deciding on the best platform for speech-to-text development involved in evaluating various options based on latency was another difficulty, along with ensuring the chat bot remained on topic with coherent responses were some of the final backend challenges.

## Accomplishments that we're proud of

We are proud of figuring out and programming an organized UI platform using an architecture and framework we had no experience with. Working together as a team and problem solving with each through issues in speech to text interpretation, prompt engineering, and training the LLM is something we feel great about.

## What we learned

We learned that pivoting to a brand new UI platform with no prior knowledge about is a challenging task, especially regarding the difficulty of having a comprehensive understanding within the duration of the hackathon. We learned how to implement OpenAI APIs and Speech Text APIs from Google Cloud, and the development behind creating speech(from user) to speech(chatbot) communication. We also learned how to divide and conquer, as well as work together on tasks that were difficult.

## What's next for Polyglot.AI

Fine tuning the LLM for a more structured response from the chat bot would make the learning experience more ideal. Improving our UIUX process and design it to be more interactive and flow more effectively. Improving the environment of the platform to make the communication purely as speech to speech.
Learning a new language doesn't happen inside an app or in front of a screen. It happens in real life.

So we created Dojo, an immersive and interactive language learning platform, powered by artificial intelligence, that supports up to 10 different languages.

Our lessons are practical. They're personalized to your everyday life, encouraging you to make connections with the world around you.

Go on a scavenger hunt and take photos of objects that match the given hint to unlock the next level. Open the visual dictionary and snap a photo of something to learn how to describe it in the language you're learning.