## Inspiration

The main focus of our project is creating opportunities for people to interact virtually and pursue their interests while remaining active. We hoped to accomplish this through a medium that people are already interested in and providing them with a tool to take that interest to the next level. From these intentions came our project- TikTok Dance Trainer.

In our previous hackathon, we gained experience with computer vision using OpenCV2 in python, and we wanted to look further in this field. Gaining inspiration from other projects that we saw, we wanted to create a project that could not only recognize hand movements but full body motion as well.

## What it does

TikTok Dance Trainer is a new Web App that enables its users to learn and replicate popular dances from TikTok. While using the app, users will receive a score in real time that gives them feedback on how well they are dancing compared to the original video. This web app is an encouraging way for beginners to hone dance skills and improve their TikTok content as well as a fun way for advanced users to compete against one another in perfecting dances.

## How we built it

To create this project, we split into teams. One team experimented with comparison metrics to compare body poses while the other built up the UI with HTML, CSS and Javascript.

The pose estimation is implemented with an open source pre-trained neural network in tensorflow called posenet. This model can pinpoint the key points on the human body such as wrists, elbows, hips, knees, and joints on the head. The two dancers each have a set of 17 joints, which are then compared to each other, frame by frame. In order to compare these arrays of coordinates, we researched various distance metrics to use such as the Euclidean Metric, Cosine Similarity, the weighted Manhattan distance, and Procrustes Analysis (Affine Transformation). Through data collection and trial and error, the cosine distance gave the best results in the end. The resulting distances were then fed into a function to map the values to viable player scores.

The UI is built up in HTML with CSS styling and Javascript to run its functions. It has a hand-drawn background and an easy-to-use design packed with function. The menu bar has a file selector for choosing and uploading a dance video to compare to. The three main cards of the UI have the reference video and live cam side by side, with pose-estimated skeletons of each in the middle to aid in matching the reference dance. The whole UI is built up in general keeping in mind ease of use, simplicity, visual appeal and functionality.

## Challenges we ran into

As a result of splitting into two teams for different parts of the project, one challenge we faced was merging the two parts. It was difficult to both combine the code but as well to connect the different parts of it, returning outputs from one part as acceptable inputs for another. Through perseverance and a lot of communication we managed to effectively merge the two parts.

## Accomplishments that we're proud of

We managed to create a clean looking app that performs the algorithm well despite the time pressure and complexity of the project. In addition, we were able to allocate time into making a presentation with a skit to tie everything together.

## What we learned

Coming into this hackathon, only one of our members was experienced in web development, but coming out, all of us four felt that we gained valuable experience and insight into the ins and outs of webpages. We learned how to effectively use Node.js to create a backend and connect it with our frontend. Along with this, we gained experience using npm and many of javascript's potpourri of packages such as browserify.

## What's next for TikTok Dance Trainer

We also looked into using Dynamic Time Warping to help with the comparison. This would help primarily when the videos were different lengths or if the dancers were slightly mismatched. However, we realized that this would not be needed if the user is dancing against the TikTok video in their own live feed. In the future, we would like to add a functionality that allows two pre-recorded videos to be compared that would then use Dynamic Time Warping.

All open source repositories/packages that were used:
[link](https://github.com/tensorflow/tfjs-models/tree/master/posenet)
[link](https://github.com/compute-io/cosine-similarity)
[link](https://github.com/GordonLesti/dynamic-time-warping)
[link](https://github.com/browserify/browserify)
[link](https://github.com/ml5js/ml5-library)
## 💡Inspiration

Gaming is often associated with sitting for long periods of time in front of a computer screen, which can have negative physical effects. In recent years, consoles such as the Kinect and Wii have been created to encourage physical fitness through games such as "Just Dance". However, these consoles are simply incompatible with many of the computer and arcade games that we love and cherish.

## ❓What it does

We came up with Motional at HackTheValley wanting to create a technological solution that pushes the boundaries of what we’re used to and what we can expect. Our product, Motional, delivers on that by introducing a new, cost-efficient, and platform-agnostic solution to universally interact with video games through motion capture, and reimagining the gaming experience.

Using state-of-the-art machine learning models, Motional can detect over 500 features on the human body (468 facial features, 21 hand features, and 33 body features) and use these features as control inputs to any video game.

Motional operates in 3 modes: using hand gestures, face gestures, or full-body gestures. We ship certain games out-of-the-box such as Flappy Bird and Snake, with predefined gesture-to-key mappings, so you can play the game directly with the click of a button. For many of these games, jumping in real-life (body gesture) /opening the mouth (face gesture) will be mapped to pressing the "space-bar"/"up" button.

However, the true power of Motional comes with customization. Every simple possible pose can be trained and clustered to provide a custom command. Motional will also play a role in creating a more inclusive gaming space for people with accessibility needs, who might not physically be able to operate a keyboard dexterously.

## 🤔 How we built it

First, a camera feed is taken through Python OpenCV. We then use Google's Mediapipe models to estimate the positions of the features of our subject. To learn a new gesture, we first take a capture of the gesture and store its feature coordinates generated by Mediapipe. Then, for future poses, we compute a similarity score using euclidean distances. If this score is below a certain threshold, we conclude that this gesture is the one we trained on. An annotated image is generated as an output through OpenCV. The actual keyboard presses are done using PyAutoGUI.

We used Tkinter to create a graphical user interface (GUI) where users can switch between different gesture modes, as well as select from our current offering of games. We used MongoDB as our database to keep track of scores and make a universal leaderboard.

## 👨‍🏫 Challenges we ran into

Our team didn't have much experience with any of the stack before, so it was a big learning curve. Two of us didn't have a lot of experience in Python. We ran into many dependencies issues, and package import errors, which took a lot of time to resolve. When we initially were trying to set up MongoDB, we also kept timing out for weird reasons. But the biggest challenge was probably trying to write code while running on 2 hours of sleep...

## 🏆 Accomplishments that we're proud of

We are very proud to have been able to execute our original idea from start to finish. We managed to actually play games through motion capture, both with our faces, our bodies, and our hands. We were able to store new gestures, and these gestures were detected with very high precision and low recall after careful hyperparameter tuning.

## 📝 What we learned

We learned a lot, both from a technical and non-technical perspective. From a technical perspective, we learned a lot about the tech stack (Python + MongoDB + working with Machine Learning models). From a non-technical perspective, we worked a lot working together as a team and divided up tasks!

## ⏩ What's next for Motional

We would like to implement a better GUI for our application and release it for a small subscription fee as we believe there is a market for people that would be willing to invest money into an application that will help them automate and speed up everyday tasks while providing the ability to play any game they want the way they would like. Furthermore, this could be an interesting niche market to help gamify muscle rehabilition, especially for children.
## Inspiration

The idea behind JustChi stemmed from a desire to improve the physical and mental well-being of seniors through accessible and engaging technology. Since everyone on the team has a relative who would benefit from the gentle exercise that is Tai Chi, we thought it would be a wonderful idea to prototype a futuristic VR game targeting the growing senior population. By combining the ancient practice of Tai Chi, known for its health benefits such as improved balance, flexibility, and stress reduction, with modern Virtual Reality (VR) and Computer Vision technology, JustChi aims to provide a unique and immersive experience that is both beneficial and enjoyable for its users.

## What it does

Think "Just Dance" but instead of high-intensity dance, our users are immersed in relaxing and peaceful VR environments where they can engage in the ancient Chinese art of Tai Chi. Once set up on the Meta Quest 2, our users can select from a variety of Tai Chi styles and soothing background environments. They are then sent to these environments where they can follow along with our prerecorded Tai Chi "master" through a game-like experience. Along the way, we provide supportive feedback and calculate the accuracy of their moves, using it to display their Tai Chi score in real-time. Through Just Chi, anyone can learn Tai Chi and reap a myriad of health benefits while being educated on ancient Chinese culture.

## How we built it

We used an open-source computer vision framework called [MediaPipe](https://developers.google.com/mediapipe) to determine our user's pose by tracking keypoints (and their coordinates) displayed along the user's body, storing and transmitting these coordinates through a Flask server. We then ran these coordinates through our pose comparison Python algorithm, which compares the coordinates of the user's keypoints with the keypoints of our Tai Chi "master" using linear algebra techniques. Then, we used Figma, HTML, CSS, JavaScript, and a VR development framework called [A-Frame](https://aframe.io/) to design our VR UI/UX and environment that is displayed to the user through the Meta Quest headset.

## Challenges we ran into

Initially, it was difficult to find a way to track keypoints while interfacing with Unity (a game engine we originally thought about using). However, we eventually found A-Frame and never looked back. It was incredibly challenging to write the pose comparison algorithm because we had trouble syncing the Tai Chi "master" recording with the user in order to compare their keypoint coordinates. Even once we implemented several different versions of the algorithm, it wasn't especially accurate at times. It was also difficult for us to design the UI/UX and environment for the Meta Quest. It was all of our first time working with VR so it was quite the learning curve for us.

## Accomplishments that we're proud of

We're proud of our functional pose comparison algorithm which we finally got to work after almost an entire day of programming, testing, and debugging. We're also very proud that we were able to have a working, enjoyable, and impactful VR game done in less than 2 days. Additionally, we are proud that we were able to find A-Frame and learn a VR web development framework that was previously completely unknown to us. Finally, we are so proud that we were able to spend the weekend with so many passionate, talented, and friendly people.

## What we learned

We learned a lot about VR development frameworks such as A-Frame, the different kinds of computer vision frameworks such as MediaPipe, the amazing art of Tai Chi, and vector similarity algorithms. Particularly, we learned that there are other VR development platforms out there (besides Unity), how to use MediaPipe, different forms of Tai Chi styles (Chen, Yang, Hao, Wu, Chen, etc), and how we can vectorize information and use linear algebra to determine similarity. Most importantly, we learned more about how prevalent depression and health issues are among the elderly, motivating us to create a fitness program that combats these issues: JustChi.

## What's next for JustChi

Our roadmap is filled with exciting developments, from introducing multiplayer capabilities to integrating heart rate monitoring for health and fitness insights. We're committed to expanding JustChi's content, offering new Tai Chi styles, and creating more immersive and personalized environments. The potential for JustChi to make a significant impact is boundless, and we're thrilled to continue this journey of innovation and wellness.