## ❓The problem at hand.

How might we help lower the barrier of entrance for **aspiring actors** to join and thrive in the entertainment industry?

## 💭 Our inspiration?

Just as others are inspired to pursue careers as engineers, doctors, or lawyers, my aspirations were shaped by watching actors perform. From a young age, witnessing their presence on Disney Channel and Netflix ignited my passion and desire to become an actor.

Due to societal pressures and the slim chances of "making it big," I was encouraged to pursue career paths deemed more sustainable and stable. Ultimately, I steered away from my Disney Channel dream.

However, there are still some out there who **have not given up**. There are ones out there who are like me, who are sitting in front of their TV, admiring the art of performing.

## 📌 Diving deeper into our problem scope

Through our research, we found an overwhelming statistic that pointed to the fact that **unemployment rates** in actors hover around 90%, and that as low as 2% of actors are able to **make a living** out of acting.

Using that, we were able to narrow down our niche and target our platform for beginners who aspire to perform center stage one day. We looked into established resources like Khan Academy, LeetCode, and Coursera as inspiration to provide purposeful and industry standard lesson plans. We believe just like any school subject like math or science, **acting is a skill that can be learned and honed onto**.

## 🎬 What does Essence do?

Essence is a platform that **trains aspiring actors from practicing to performing**. Users are presented with real life scenes to watch and practice by mimicking. They are able to record themselves scripting the scene and get an **instant score** and **personalized feedback** to improve their skills. Users can compare side by side how the actor figure performed versus their take of the same scene. They are able to review all their takes and view their progress through their acting journey!

## 🛠️ How we built it

Our team started by brainstorming the core features we wanted Essence to have, focusing on user engagement and practical learning tools. We then designed the user interface and user experience in Figma, ensuring a sleek and intuitive design. On the backend, we set up the server, database, and APIs needed to support our platform's functionalities. For the frontend, we developed the client-side interface, focusing on creating a responsive and user-friendly environment. Finally, we integrated the frontend and backend, ensuring seamless interaction between components.

## 🚦Challenges we ran into

It was challenging to agree on which features were essential for our initial launch. We encountered several technical issues, especially related to real-time feedback and performance comparisons. Once many components were implemented, it became difficult to make changes to the overall architecture without disrupting existing features.

## 🖍️ Accomplishments that we're proud of

We are proud of the intuitive and visually appealing user interface we created, and achieving a fully functional platform that effectively helps users improve their acting skills.

## 🎤 What we learned

Our team gained experience with **new technologies and design principles**. We enhanced our version control skills, making collaboration more efficient. We learned how to work better as a team, leveraging each other’s strengths and improving our communication.

## → What's next for Essence

As Essence provides easy to use resources for actors who are just starting out, we want to increase the amount of features to help actors of all levels. Think generative prompts to practice expressing niche emotions and involving the mentorship of admirable actors. Overall, we strive to provide an ever growing and supportive community for actors of all levels to come together!
## Inspiration

In a world where education has become increasingly remote and reliant on online platforms, we need human connection **more than ever**. Many students often find it difficult to express their feelings without unmuting themselves and drawing unwanted attention. As a result, teachers are unaware of how their students are feeling and if the material is engaging. This situation is especially challenging for students who struggle with communicating their feelings–such as individuals with autism, selective mutism, social anxiety, and more.

We want to help **bridge this gap** by creating a tool that will both enable students to express themselves with less effort and enable teachers to understand and respond to their overall needs.

We strongly believe in the importance of **accessibility in education** and supplementing human connection, because at the end of the day, humans are all social beings.

## What it does

Our application helps measure the general emotions of participants in a video meeting, displaying a stream of emojis representing up to **80 different emotions**. We periodically sample video frames from all participants with their cameras on at 10-second intervals, feeding this data into **Hume’s Expression Management API** to identify the most prominent expressions. From this, we generate a composite view of the general sentiment using a custom weighted algorithm.

Using this aggregated sentiment data, our frontend displays the most frequent emotions with their corresponding emojis on the screen. This way, hosts can adapt their teaching to the general sentiment of the classroom, while students can share how they’re feeling without having to experience the social anxiety that comes with typing a message in the chat or sharing a thought out loud.

## How we built it

We leveraged **LiveKit** to create our video conference infrastructure and **Vercel** to deploy our application. We also utilized **Supabase Realtime** as our communication protocol, forwarding livestream data from clients per room and saving that data to Supabase Storage.

Our backend, implemented with **FastAPI**, interfaces with the frontend to pull this data from Supabase and feed the captured facial data into Hume AI to detect human emotions.

The results are then aggregated and stored back into our Supabase table. Our frontend, built with **Next.js** and styled with **Tailwind CSS**, listens to real-time event triggers from Supabase to detect changes in the table.

From this, we’re able to display the stream of emotions in **near real-time**, finally delivering aggregated emotion data as a light-hearted fun animation to keep everyone engaged!

## Challenges we ran into

* Livekit Egress has limited documentation
* Coordination of different parts using Supabase Realtime
* Hume AI API
* First-time Frontenders
* Hosting our backend thru Vercel (lots of config)

## Accomplishments that we're proud of

* Livekit real time streaming video conference
* Streaming video data to Hume Supabase Realtime
* Emoji animation using Framer Motion
* Efficient scoring algorithm using heaps

## What we learned

We learned how to use a lot of new tools and frameworks such as Next.js and Supabase as it was some of our members' first time doing full-stack software engineering. From our members all the way from SoCal and the East Coast, we learned how to ride the BART, and we all learned LiveKit for live streaming and video conferencing.

## What's next for Moji

We see the potential of this tool in a **wide variety of industries** and have other features in mind that we want to implement. For example, we can focus on enhancing this tool to help streamers with any kind of virtual audience by:

* Implementing a dynamic **checklist** that generates to-dos based on questions or requests from viewers.

This can benefit teachers in providing efficient learning to their studies or large entertainment streamers in managing a fast-moving chat. This can also be extended to eCommerce, as livestream shopping requires sellers to efficiently navigate their chat interactions.

* Using Whisper for **real-time audio speech recognition** to automatically check off answered questions.

This provides a hands-free way for streamers to meet their viewers’ requests without having to look extensively through chat. This is especially beneficial for the livestream shopping industry as sellers are typically displaying items while reading messages

* Using **RAG** to store answers to previously asked questions and using this data to answer any future questions.

This can be a great way to save time for streamers from answering repeated questions.
Enhancing video recognition capabilities to identify more complex interactions and objects in real-time.
With video recognition, we can lean even heavier into the eCommerce industry, identifying what type of products sellers are displaying and providing a hands-free and AI enhanced way of managing their checklist of requests.

* Adding **integrations** with other streaming platforms to broaden its applicability and improve the user experience.

The possibilities are endless and we’re excited to see where Moji can go! We hope that Moji can bring a touch of humanity and help us all stay connected and engaged in the digital world.
## Inspiration

We were inspired by hard working teachers and students. Although everyone was working hard, there was still a disconnect with many students not being able to retain what they learned. So, we decided to create both a web application and a companion phone application to help target this problem.

## What it does

The app connects students with teachers in a whole new fashion. Students can provide live feedback to their professors on various aspects of the lecture, such as the volume and pace. Professors, on the other hand, get an opportunity to receive live feedback on their teaching style and also give students a few warm-up exercises with a built-in clicker functionality.
The web portion of the project ties the classroom experience to the home. Students receive live transcripts of what the professor is currently saying, along with a summary at the end of the lecture which includes key points. The backend will also generate further reading material based on keywords from the lecture, which will further solidify the students’ understanding of the material.

## How we built it

We built the mobile portion using react-native for the front-end and firebase for the backend. The web app is built with react for the front end and firebase for the backend. We also implemented a few custom python modules to facilitate the client-server interaction to ensure a smooth experience for both the instructor and the student.

## Challenges we ran into

One major challenge we ran into was getting and processing live audio and giving a real-time transcription of it to all students enrolled in the class. We were able to solve this issue through a python script that would help bridge the gap between opening an audio stream and doing operations on it while still serving the student a live version of the rest of the site.

## Accomplishments that we’re proud of

Being able to process text data to the point that we were able to get a summary and information on tone/emotions from it. We are also extremely proud of the

## What we learned

We learned more about React and its usefulness when coding in JavaScript. Especially when there were many repeating elements in our Material Design. We also learned that first creating a mockup of what we want will facilitate coding as everyone will be on the same page on what is going on and all thats needs to be done is made very evident. We used some API’s such as the Google Speech to Text API and a Summary API. We were able to work around the constraints of the API’s to create a working product. We also learned more about other technologies that we used such as: Firebase, Adobe XD, React-native, and Python.

## What's next for Gradian

The next goal for Gradian is to implement a grading system for teachers that will automatically integrate with their native grading platform so that clicker data and other quiz material can instantly be graded and imported without any issues. Beyond that, we can see the potential for Gradian to be used in office scenarios as well so that people will never miss a beat thanks to the live transcription that happens.