# CourseAI: AI-Powered Personalized Learning Paths

## Inspiration

CourseAI was born from the challenges of self-directed learning in our information-rich world. We recognized that the issue isn't a lack of resources, but rather how to effectively navigate and utilize them. This inspired us to leverage AI to create personalized learning experiences, making quality education accessible to everyone.

## What it does

CourseAI is an innovative platform that creates personalized course schedules on any topic, tailored to the user's time frame and desired depth of study. Users input what they want to learn, their available time, and preferred level of complexity. Our AI then curates the best online resources into a structured, adaptable learning path. Key features include:

* AI-driven content curation from across the web
* Personalized scheduling based on user preferences
* Interactive course customization through an intuitive button-based interface
* Multi-format content integration (articles, videos, interactive exercises)
* Progress tracking with checkboxes for completed topics
* Adaptive learning paths that evolve based on user progress

## How we built it

We developed CourseAI using a modern, scalable tech stack:

* Frontend: React.js for a responsive and interactive user interface
* Backend Server: Node.js to handle API requests and serve the frontend
* AI Model Backend: Python for its robust machine learning libraries and natural language processing capabilities
* Database: MongoDB for flexible, document-based storage of user data and course structures
* APIs: Integration with various educational content providers and web scraping for resource curation

The AI model uses advanced NLP techniques to curate relevant content, and generate optimized learning schedules. We implemented machine learning algorithms for content quality assessment and personalized recommendations.

## Challenges we ran into

1. API Cost Management: Optimizing API usage for content curation while maintaining cost-effectiveness.
2. Complex Scheduling Logic: Creating nested schedules that accommodate various learning styles and content types.
3. Integration Complexity: Seamlessly integrating diverse content types into a cohesive learning experience.
4. Resource Scoring: Developing an effective system to evaluate and rank educational resources.
5. User Interface Design: Creating an intuitive, button-based interface for course customization that balances simplicity with functionality.

## Accomplishments that we're proud of

1. High Accuracy: Achieving a 95+% accuracy rate in content relevance and schedule optimization.
2. Elegant User Experience: Designing a clean, intuitive interface with easy-to-use buttons for course customization.
3. Premium Content Curation: Consistently sourcing high-quality learning materials through our AI.
4. Scalable Architecture: Building a robust system capable of handling a growing user base and expanding content library.
5. Adaptive Learning: Implementing a flexible system that allows users to easily modify their learning path as they progress.

## What we learned

This project provided valuable insights into:

* The intricacies of AI-driven content curation and scheduling
* Balancing user preferences with optimal learning strategies
* The importance of UX design in educational technology
* Challenges in integrating diverse content types into a cohesive learning experience
* The complexities of building adaptive learning systems
* The value of user-friendly interfaces in promoting engagement and learning efficiency

## What's next for CourseAI

Our future plans include:

1. NFT Certification: Implementing blockchain-based certificates for completed courses.
2. Adaptive Scheduling: Developing a system for managing backlogs and automatically adjusting schedules when users miss sessions.
3. Enterprise Solutions: Creating a customizable version of CourseAI for company-specific training.
4. Advanced Personalization: Implementing more sophisticated AI models for further personalization of learning paths.
5. Mobile App Development: Creating native mobile apps for iOS and Android.
6. Gamification: Introducing game-like elements to increase motivation and engagement.
7. Peer Learning Features: Developing functionality for users to connect with others studying similar topics.

With these enhancements, we aim to make CourseAI the go-to platform for personalized, AI-driven learning experiences, revolutionizing education and personal growth.
## Inspiration

The inspiration behind Mentis emerged from a realization of the vast potential that a personalized learning AI platform holds in transforming education. We envisioned an AI-driven mentor capable of adapting to individual learning styles and needs, making education more accessible, engaging, and effective for everyone. The idea was to create an AI that could dynamically update its teaching content based on live user questions, ensuring that every learner could find a path that suits them best, regardless of their background or level of knowledge.

We wanted to build something that encompasses the intersection of: accessibility, interactivity and audiovisual.

## What it does

Mentis is an AI-powered educational platform that offers personalized learning experiences across a wide range of topics. It is able to generate and teach animated lesson plans with both visuals and audio, as it generates checkpoint questions for the user and listens to the questions of its users and dynamically adjusts the remainder of the teaching content and teaching methods to suit their individual learning preferences. Whether it's mathematics, science, or economics, Mentis provides tailored guidance, ensuring that users not only receive answers to their questions but also a deep understanding of the subject matter.

## How we built it

At its core, a fast API backend powers the intelligent processing and dynamic delivery of educational content, ensuring rapid response to user queries. This backend is complemented by our use of advanced Large Language Models (LLMs), which have been fine-tuned to understand a diverse range of educational topics and specialize in code generation for the best animation, enhancing the platform's ability to deliver tailored learning experiences.

We curated a custom dataset in order to leverage LLMs to the fullest and reduce errors in both script and code generation. Using our curated datasets, we were able to fine-tune models using MonsterAPI tailoring our LLMs and improve accuracy.. We implemented several API calls to ensure a smooth and dynamic operation of our platform, for general organization of the lesson plan, script generation, audio generation with ElevenLabs, and code generation for the manim library we utilize to create the animations on our front end in Bun and Next.js.

[Fine tuned open source model](https://huggingface.co/generaleoley/mixtral-8x7b-manim-lora/tree/main)

[Curated custom dataset](https://huggingface.co/datasets/generaleoley/manim-codegen)

## Challenges we ran into

Throughout the development of Mentis, we encountered significant challenges, particularly in setting up environments and installing various dependencies. These hurdles consumed a considerable amount of our time, persisting until the final stages of development.

Every stage of our application had issues we had to address: generating dynamic sections for our video scripts, ensuring that the code is able to execute the animation, integrating the text-to-speech component to generate audio for our educational content all introduced layers of complexity, requiring precise tuning and a lot of playing with to set up.

The number of API calls needed to fetch, update, and manage content dynamically, coupled with ensuring the seamless interaction between the user and our application, demanded a meticulous approach. We found ourselves in a constant battle to maintain efficiency and reliability, as we tried to keep our latency low for practicality and interactivity of our product.

## Accomplishments that we're proud of

Despite the setbacks, we are incredibly proud of:

* Technical Overcomes: Overcoming significant technical hurdles, learning from them and enhancing our problem-solving capabilities.
* Versatile System: Enabling our platform to cover a broad range of topics, making learning accessible to everyone.
* Adaptive Learning: Developing a system that can truly adapt to each user's unique learning style and needs.
* User-Friendly UI: Creating a user-friendly design and experience keeping our application as accessible as possible.
* API Management: Successfully managing numerous API calls, we smoothed the backend operation as much as possible for a seamless user experience.
* Fine tuned/Tailored Models: Going through the full process of data exploration & cleaning, model selection, and configuring the fine-tuned model.

## What we learned

Throughout the backend our biggest challenge and learning point was the setup, coordination and training of multiple AI agents and APIs.

For all of us, this was our first time fine-tuning a LLM and there were many things we learned through this process such as dataset selection, model selection, fine-tuning configuration. We gained an appreciation for all the great work that was being done by the many researchers. With careful tuning and prompting, we were able to greatly increase the efficiency and accuracy of the models.

We also learned a lot about coordinating multi-agent systems and how to efficiently have them run concurrently and together. We tested many architectures and ended up settling for one that would optimize first for accuracy then for speed. To accomplish this, we set up an asynchronous query system where multiple “frames” can be generated at once and allow us not to be blocked by cloud computation time.

## What's next for mentis.ai

Looking ahead, Mentis.ai has exciting plans for improvement and expansion:

**Reducing Latency:** We're committed to enhancing efficiency, aiming to minimize latency further and optimize performance across the platform.

**Innovative Features:** Given more time, we plan to integrate cutting-edge features, like using HeyGen API to create natural videos of personalized AI tutors, combining custom images, videos, and audio for a richer learning experience.

**Classroom Integration:** We're exploring opportunities to bring Mentis into classroom settings, testing its effectiveness in a real-world educational environment and tailoring its capabilities to support teachers and students alike.
## Inspiration

Across the globe, a critical shortage of qualified teachers poses a significant challenge to education. The average student-to-teacher ratio in primary schools worldwide stands at an alarming **23:1!** In some regions of Africa, this ratio skyrockets to an astonishing **40:1**. [Research 1](https://data.worldbank.org/indicator/SE.PRM.ENRL.TC.ZS) and [Research 2](https://read.oecd-ilibrary.org/education/education-at-a-glance-2023_e13bef63-en#page11)

As populations continue to explode, the demand for quality education has never been higher, yet the *supply of capable teachers is dwindling*. This results in students receiving neither the attention nor the **personalized support** they desperately need from their educators.

Moreover, a staggering **20% of students** experience social anxiety when seeking help from their teachers. This anxiety can severely hinder their educational performance and overall learning experience. [Research 3](https://www.cambridge.org/core/journals/psychological-medicine/article/much-more-than-just-shyness-the-impact-of-social-anxiety-disorder-on-educational-performance-across-the-lifespan/1E0D728FDAF1049CDD77721EB84A8724)

While many educational platforms leverage generative AI to offer personalized support, we envision something even more revolutionary. Introducing **TeachXR—a fully voiced, interactive, and hyper-personalized AI** teacher that allows students to engage just like they would with a real educator, all within the immersive realm of extended reality.

*Imagine a world where every student has access to a dedicated tutor who can cater to their unique learning styles and needs. With TeachXR, we can transform education, making personalized learning accessible to all. Join us on this journey to revolutionize education and bridge the gap in teacher shortages!*

## What it does

**Introducing TeachVR: Your Interactive XR Study Assistant**

TeachVR is not just a simple voice-activated Q&A AI; it’s a **fully interactive extended reality study assistant** designed to enhance your learning experience. Here’s what it can do:

* **Intuitive Interaction**: Use natural hand gestures to circle the part of a textbook page that confuses you.
* **Focused Questions**: Ask specific questions about the selected text for summaries, explanations, or elaborations.
* **Human-like Engagement**: Interact with TeachVR just like you would with a real person, enjoying **milliseconds response times** and a human voice powered by **Vapi.ai**.
* **Multimodal Learning**: Visualize the concepts you’re asking about, aiding in deeper understanding.
* **Personalized and Private**: All interactions are tailored to your unique learning style and remain completely confidential.

### How to Ask Questions:

1. **Circle the Text**: Point your finger and circle the paragraph you want to inquire about.
2. **OK Gesture**: Use the OK gesture to crop the image and submit your question.

### TeachVR's Capabilities:

* **Summarization**: Gain a clear understanding of the paragraph's meaning. TeachVR captures both book pages to provide context.
* **Examples**: Receive relevant examples related to the paragraph.
* **Visualization**: When applicable, TeachVR can present a visual representation of the concepts discussed.
* **Unlimited Queries**: Feel free to ask anything! If it’s something your teacher can answer, TeachVR can too!

### Interactive and Dynamic:

TeachVR operates just like a human. You can even interrupt the AI if you feel it’s not addressing your needs effectively!

## How we built it

**TeachXR: A Technological Innovation in Education**

TeachXR is the culmination of advanced technologies, built on a microservice architecture. Each component focuses on delivering essential functionalities:

### 1. Gesture Detection and Image Cropping

We have developed and fine-tuned a **hand gesture detection system** that reliably identifies gestures for cropping based on **MediaPipe gesture detection**. Additionally, we created a custom **bounding box cropping algorithm** to ensure that the desired paragraphs are accurately cropped by users for further Q&A.

### 2. OCR (Word Detection)

Utilizing **Google AI OCR service**, we efficiently detect words within the cropped paragraphs, ensuring speed, accuracy, and stability. Given our priority on latency—especially when simulating interactions like pointing at a book—this approach aligns perfectly with our objectives.

### 3. Real-time Data Orchestration

Our goal is to replicate the natural interaction between a student and a teacher as closely as possible. As mentioned, latency is critical. To facilitate the transfer of image and text data, as well as real-time streaming from the OCR service to the voiced assistant, we built a robust data flow system using the **SingleStore database**. Its powerful real-time data processing and lightning-fast queries enable us to achieve sub-1-second cropping and assistant understanding for prompt question-and-answer interactions.

### 4. Voiced Assistant

To ensure a natural interaction between students and TeachXR, we leverage **Vapi**, a natural voice interaction orchestration service that enhances our feature development. By using **DeepGram** for transcription, **Google Gemini 1.5 flash model** as the AI “brain,” and **Cartesia** for a natural voice, we provide a unique and interactive experience with your virtual teacher—all within TeachXR.

## Challenges we ran into

### Challenges in Developing TeachXR

Building the architecture to keep the user-cropped image in sync with the chat on the frontend posed a significant challenge. Due to the limitations of the **Meta Quest 3**, we had to run local gesture detection directly on the headset and stream the detected image to another microservice hosted in the cloud. This required us to carefully adjust the size and details of the images while deploying a hybrid model of microservices. Ultimately, we successfully navigated these challenges.

Another difficulty was tuning our voiced assistant. The venue we were working in was quite loud, making background noise inevitable. We had to fine-tune several settings to ensure our assistant provided a smooth and natural interaction experience.

## Accomplishments that we're proud of

### Achievements

We are proud to present a complete and functional MVP! The cropped image and all related processes occur in **under 1 second**, significantly enhancing the natural interaction between the student and **TeachVR**.

## What we learned

### Developing a Great AI Application

We successfully transformed a solid idea into reality by utilizing the right tools and technologies.

There are many excellent pre-built solutions available, such as **Vapi**, which has been invaluable in helping us implement a voice interface. It provides a user-friendly and intuitive experience, complete with numerous settings and plug-and-play options for transcription, models, and voice solutions.

## What's next for TeachXR

We’re excited to think of the future of **TeachXR** holds even greater innovations! we’ll be considering\**adaptive learning algorithms*\* that tailor content in real-time based on each student’s progress and engagement.

Additionally, we will work on integrating **multi-language support** to ensure that students from diverse backgrounds can benefit from personalized education. With these enhancements, TeachXR will not only bridge the teacher shortage gap but also empower every student to thrive, no matter where they are in the world!