**DEMO VIDEO**: <https://drive.google.com/drive/folders/1Eh3pVSfi8KzsKQ5bC8A-uBp98ogV6XRr>
The video was not able to upload to YouTube or other platforms so we have attached a Google Drive link here (Devpost would not let us put it in the video field).

## Inspiration

Most students don’t encounter machine learning until college. However, this typical track of Calculus 2 > Machine Learning may be outdated. With the development of powerful machine learning tools like IntersystemsML, complex mathematics is no longer necessary for a conceptual understanding of machine learning.

**We make the bold claim that kids – even middle schoolers – can and should be exposed to machine learning concepts.** With this goal, Whisker Workshops sets out to preserve precious machine learning insights while replacing formidable blocks of code and equations with cute hand-drawn graphics and games.

## What it does

Whisker Workshops consists of two main playground sections: one allows students to interactively build a neural network to represent the nonlinear exclusive or (XOR) gate while the other allows users to explore various dimensionality reduction levels for principal component analysis (PCA). We designed the UI to be cute and hand-drew graphics down to the buttons, aiming to foster a friendly and welcoming environment for beginners.

For the XOR gate playground, the truth table for the logic gate is given (no prior experience is assumed) — the goal is then to create a network to reproduce XOR. The playground is very visual: the user can see the entire neural network in real-time while updating neurons and edges. Edge weights can be updated by clicking on them, and the thickness of an edge visualizes its corresponding weight; the number of neurons is adjustable through a slider. Then, the player can run the network on different inputs and see how well they do: the task is essentially presented as a game of sorts!

The PCA playground (also known as “More Isn’t Always Better”) follows a similar visual philosophy: users can change the dimensionality of input data (Fashion MNIST dataset, images of clothes and accessories) and visualize how well the model performs through an interactive chart. Our friendly cat mentor also guides us through the entire process and explains why we observe the results we do!

## How we built it

All of the graphics and artwork in the final product were drawn by our team and incorporated into the website frontend built in React.JS and CSS.

The PCA playground was built by first conducting principal component analysis in Python on the Fashion MNIST dataset for a variety of output dimensionalities. We then used InterSystems IntegratedML to train a machine learning model for each of the resulting datasets. Through this process, we were delightfully surprised by the versatility of InterSystems IntegratedML’s capabilities. Not only was the model able to predict the 2-way XOR task, a classic non-linearity test, but it was also able to perform quite well on classifying MNIST fashion pieces, with no knowledge of what the input was.

The resulting accuracy data was then visualized through Chart.JS, a data visualization library in Javascript. The cat animations and dialogue were all created in CSS.

The XOR gate playground was designed primarily in React.JS, using a lot of different state logic and hooks to create the interactive neural network editor and interface for running the model. Due to the highly custom nature of the project, we manually implemented forward propagation for the network. All styling was done in CSS.

## Challenges we ran into

It was often difficult to create certain visual designs, effects and features with CSS: for example, the slider that demonstrates the results of Principal Component Analysis for different output dimensions. This was especially the case for importing our custom graphics and artwork into our project. This improved our debugging skills significantly and also taught us how to better utilize documentation to learn.

Another challenge we faced was navigating and running models on IntegratedML. We are very grateful for Thomas from Intersystems for his exceedingly helpful advice, allowing us to successfully train and validate the models we needed.

## Accomplishments that we're proud of

We’re proud of having built a functional application with real-world value during this hackathon. As a team, we believe that Whisker Workshop isn’t just a toy project but a starting point for future AI education initiatives aimed at a younger audience. Our team is also proud to have explored and combined skills from a lot of different fields including machine learning, web development, data analysis, and graphic design.

Another accomplishment that we feel proud of is the fact that we incorporated the InterSystems IntegratedML framework into our project as this took us a lot of effort and research.

## What we learned

We learned how to make a stylish and customized UI and utilize CSS to its fullest extent; along with incorporating custom graphic design, we learned to create a product that would appeal to our target audience: children without technical knowledge interested in exploring machine learning. Thinking about our target audience was a skill we improved a lot throughout TreeHacks as we continually aimed to look at our product from users’ perspective.

Using InterSystems IntegratedML also taught us how to effectively use external tools to help build our product. Through the process, we had to read a lot of documentation and learn to understand the new paradigm presented by this specific system — this is a skill that is essential as programmers always use different tools and technologies to assist them.

We were also puzzled as to how to efficiently run the machine learning algorithm in real time after each update made by the user without taking a toll on the user’s computer or taking too long. We learned how to custom-build a machine learning algorithm without the conventional tools such as Tensorflow or Pytorch, which would be too inefficient to run each time. We eventually accomplished this by extracting only the features essential to the differentiation task with principal component analysis, and then coding up our own dense network.

## What's next for Whisker Workshop

We plan to expand this concept of abstractified ML to middle school and high school students who don’t have access to machine learning education. We hope to promote Whisker Workshop to elementary and middle school teachers, who can encourage their students to explore machine learning in their free time.

One extension is a playground for kids to create a multi-layer deep learning model that will perform image classification. For each layer, they will have options to input a dense or convolutional layer and sliders to tune hyperparameters — observing how different settings impact accuracy in this gamified and interactive environment will allow students to develop intuition about deep learning concepts (e.g. network architecture, activations, etc.).
### FOR EVALUATORS/SPONSORS:

Scroll down for a handy guide to navigating our repository and our project's assets.

## 💥 How it all started

Every student learns differently. Traditional educational approaches, while effective for some, often fail to cater to the diverse needs and paces of individual learners. Even traditional AI learning is often one-sided lecturing that fails to properly engage the student in actual learning. This realization sparked a quest for more personalized and adaptive teaching strategies.

The capability of AI to analyze and interpret human facial expressions in real-time presented a unique opportunity to improve on today’s sparse learning environment. We envisioned a system that could monitor students' engagement levels, emotional responses, and cognitive states during lessons, allowing for dynamically changing teaching methods.

By leveraging Hume.AI’s expression detection, we aimed to create a feedback loop where we could have an AI educator receive immediate and actionable information about their students' learning experiences. This creates a seamless experience where the user does not even need to inform the AI about whether the teaching approach is right for them—they simply can focus on learning, while letting us do the rest.

The problems encountered by the rigid, traditional teaching methods:

* Lack of Personalization: One-size-fits-all approach fails to address individual learning styles and paces, disproportionately affecting low-income and neurodivergent learners.
* Student Disengagement: Inability to maintain student interest and motivation due to uniform delivery methods.
* Limited Feedback: Current education practices struggle to gauge real-time understanding and emotional responses of all students.
* Underutilized Potential: Advanced students may not be sufficiently challenged, leading to boredom and disengagement.
* Teacher Workload: High demand on teachers to adapt lessons manually for diverse classrooms without sufficient support.
* Resistance to Change: Institutional inertia and reluctance to adopt new teaching technologies and methodologies.
* Rigid Schedules: Students must choose between real-time teaching with instant feedback (that they must attend at certain times) or asynchronous learning that does not guarantee instant feedback (as the instructor is not always online)

## 📖 What it does

To address these challenges, we developed **lock in**, an app designed to allow students of any age or level to learn the way that is best for them.

Here’s how **lock in** leverages advanced technologies to serve its mission:

* Personalized Learning Assistants: By leveraging Hume’s Empathic Voice Interface, we are able to provide students the ability to learn from and converse with an interactive AI assistant that leverage RLHF (reinforcement learning through human feedback) techniques to adjust its teaching methods based on the students facial and vocal expressions.
* Inclusive Assistant for Neurodivergent Learners: **lock in** was designed with neurodivergent learners in mind (if not at the forefront). Those with special needs in education (especially in developing parts of the world) often don’t have proper infrastructure for their learning. We especially aim to target those with ADHD, with our AI constantly watching and innovating on its education in order to keep those with shorter attention spans engaged in learning.
* Intelligent Curriculum Generation: By integrating Mathpix and Claude 3.5 through Amazon Bedrock, **lock in** is able to generate a detailed curriculum to be taught by the AI learning assistant. In doing so, students can easily supplement their in-class learning with a curriculum that is based on their current learning.
* Active Learning Encouragement: We designed our model to not simply lecture the student, but involve the student in the learning process by presenting them examples to grapple with and inviting them to connect concepts. Passive consumption only makes students feel like they’re learning, but results in less learning overall¹. We focus on inviting the student to actively grapple with material.
**lock in** is more than just an education app; it's a movement towards personalized, efficient learning. Through the use of advanced technology such as React, FastAPI, AWS, Hume, and Claude, we aim to create a world in which learning is fun for everyone.

## 🔧 How we built it

To address the challenges of creating personalized education with our app, **lock in**, we've woven together a fabric of cutting-edge software technologies, each selected for its ability to enhance the app's functionality and user experience. Here's how each technology contributes to **lock in**:

### Technologies:

* React: Powers the user interface, offering a smooth and pleasing experience
* FastAPI: Provides the backbone for the backend, handling requests efficiently and ensuring scalability to support a growing user base.
* AWS Bedrock (Claude 3.5 Sonnet): Used to give more context about syllabi, especially when they contain images and diagrams that cannot be processed by Mathpix. This allows **lock in** to more easily understand what it’s looking at. Claude also creates comprehensive lesson plans and lesson topics, which are then handed to EVI for specifics.
* Hume’s EVI (powered by Claude 3.5 Sonnet): Provides speech to speech voice chatting capabilities through EVI (Empathic Voice Interface), taking into account users’ voice tones and emotions. This allows EVI to learn how to better teach on the fly and change teaching styles if it detects boredom or other negative emotions. This switching of styles can be especially useful for neurodivergent students who may have trouble staying on target. EVI handles the specifics of teaching each lesson, handed to it by Claude on Bedrock.
* Hume’s Expression Measurement API: Provides the ability for EVI to read facial expressions and emotions, giving even more emotional context. EVI will respond to speech in a more nuanced way, with even more information than it usually has by itself.
* Mathpix: Extracting the text from syllabi, allowing it to be passed into Claude to generate a curriculum. This allows the user to easily and accurately import pre-existing lesson plans.

Learning with AI today is currently extremely hard and time consuming. But, with the integration of these emotionally aware technologies into **lock in**, we can create a more proactive and sensitive learning environment, suitable for those young and old, as well as those neurotypical and neurodivergent.

Our AI assistant takes more control of the curriculum, rather than taking a backseat and requiring the student to come up with prompts. React and FastAPI ensure a robust and efficient app infrastructure, while Mathpix and the multimodal Claude 3.5 on AWS Bedrock give **lock in** the ability to understand images and create curriculums Lastly, Hume’s EVI (powered by Claude) and Expression Measurement API do the emotional heavy lifting. This makes **lock in** not just a lifeless learning tool, but a supportive and initiative-taking academic companion. This synergy of technologies makes **lock in** a powerful platform for individuals of all learning styles and needs, taking in human feedback and approaching change in a more human way.

## 🚩 Challenges we ran into

Information Integration: **lock in** records and processes data from multiple sources, including, but not limited to, user speech, syllabi, and user-drawn visual resources. It was a huge challenge to manage our data, creatively integrate them into different services, and process them to quickly provide Samantha’s seamless, and human-like responses.

Prompt Engineering: Our control over model input and output is fairly limited, meaning that we have to get creative with things like config settings and context. Most of our ability to specify behavior lies in our system prompt, however. This creates a nearly bottomless creative task of prompting our teaching assistant to educate our users as best as possible. This is, of course, made even more complicated by the unique personalities and learning styles of different people. We strive to capture all learners, but there is no “one-size-fits-all” prompt.

## 🏆 Accomplishments that we're proud of

Fully implementing all the major features we planned to when we first started brainstorming
Integrating both vocal conversations/vocal expression (EVI) and facial expression (Expression Measurement API) into one seamless product, which has not yet been fully explored

We improved on Learning with AI in the following ways:

1. Typical AI chatbot learning experiences revolves around students needing to take the initiative to split a potentially complex topic up into parts themselves and tracking progress on them. Our lesson-based learning helps students who prefer learning "on-rails", giving a more structured learning experience that is less daunting to those who are unfamiliar with the topics they are asking about. This makes it easier to stay on task, track progress, and understand how the current lesson fits into the topic as a whole.
2. AI chatbots revolve around inputs and outputs being types. This results in stilted, slow experiences that don't quite feel "real" to a user. **lock in** features speech input and output, giving a more quick, conversational experience. This is also a more natural way of communicating, allowing users to properly immerse themselves in learning as if they were talking to a tutor or teacher.
3. Most LLMs require a good bit of calibration and user input in order to receive the desired result. Most casual users are either not aware or not skilled at this, and thus they are not able to extract full potential from their learning. Samantha continuously tracks emotions through speech prosody and facial expressions, and will offer a change of gears to the student if she detects something isn't quite working.

## 📝 What we learned

Nicholas: I learned a lot about the strengths and limits of modern multi-modal LLMs like Claude 3.5. After experimenting with different prompts and tasks, I feel more confident on how I would use them in the future and I am excited to see how new models push the boundaries.

Leon: Participating in this AI hackathon showed me the unlimited possibilities of AI, and how it can improve so much in the world. Working on this project had a lot of various struggles, but ultimately, building together with my team was an amazing experience that allowed me to learn so much.

Ryan: This AI Hackathon was my first ever hackathon experience, so getting a hang of the process and conventions was a very novel experience. I got a far better sense of how applications are actually tied together.

Alex: I had tons of fun figuring out how to integrate all of these completely different technologies into one seamless application. It was a huge challenge to manage our data and figure out if and how things should connect to each other. This is the most data heavy project I have ever worked on, and I had a blast doing so!

## ✈️ What's next for **lock in**

Just like how a great trip has a great itinerary, we envision **lock in**’s future plans in phases.

### Phase 1: Solidifying Our Roots

Phase 1 involves solidifying the use of the whiteboard, as well as having Samantha remember specific learning characteristics of students. We also want to improve our UI to create a more streamlined and user-friendly experience.

### Phase 2: Branching Out

Phase 2 involves improving Samantha’s specific subject capabilities, especially her skills in mathematics. We may experiment with other image analysis strategies, using the multimodal Claude for understanding diagrams and Mathpix’s OCR capabilities for reading text. We also want to give her image generation capabilities to supplement explanations with visual examples and close the loop on image to image communication.

### Phase 3: Take Off

Phase 3 involves partnering with established online education companies like Brilliant.org or Curiosity, improving their services and reaching a larger community of those who love to learn.

## 📋 Evaluator's Guide to **lock in**

Intended for judges, however the viewing public is welcome to take a look.
Hey! We wanted to make this guide in order to help provide you further information on our implementations of certain programs and provide a more in-depth look to cater to both the viewing audience and evaluators like yourself.

### SPONSOR SERVICES WE HAVE USED THIS HACKATHON

* Hume
* AWS Bedrock (Claude 3.5)
* Mathpix

### REFERENCE:

1: Deslauriers L, McCarty LS, Miller K, Callaghan K, Kestin G. Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proc Natl Acad Sci U S A. 2019 Sep 24;116(39):19251-19257. doi: 10.1073/pnas.1821936116. Epub 2019 Sep 4. PMID: 31484770; PMCID: PMC6765278.
## Inspiration

Nowadays, unless you've been living under a rock, you've definitely heard of the fancy computer science buzzwords, including "machine learning", or "neural networks". Unless you have incredible resources or motivation, these words will continue to mystify you until you pour hours into understanding the concepts and mechanics behind the principle of machine learning (Trust us, we've been there and done that). Our goal is to come up with an application that gives users the ability to change a few parameters here and there, giving them a high-level overview of machine learning that is nevertheless quite informative.

## What it does

Our application allows you to build your own neural network at a high level, giving users the ability to control the type and number of layers in the network in each run of the program. The application allows you to submit your own data to train and test on, then uses TensorFlow to build several informative graphs on the neural network as it progresses through the stages of forward and back-propagation.

## How we built it

We built the framework of the application using ReactJS and the main page of the application using Canvas (vanilla Javascript). The machine learning is done with TensorFlow JS.

## Challenges we ran into

We ran into many problems with integration, especially between the vanilla JS with React, and different TensorFlow JS functions.

## Accomplishments that we're proud of

Designing the scratch for ML! We needed to integrate many different ideas to get this project to where it is, but we learned a lot of different tools along the way. The drag and drop mechanism is simple to use, and the graphs provide an idea of how ML algorithms train without the burden of having to code them.

## What we learned

Our team members came together with different levels of technical experience, but we all came out with a much better grasp and experience with React, TensorFlow, and building full-stack applications.

## What's next for ML Outreach

We plan to build more useful and cool visualizations as well as more options in machine learning, such as linear or logistic classifiers, or even unsupervised learning algorithms.