## Inspiration

I've always wanted to learn to DANCE. But dance teachers cost money and I'm a bad dancer :( We made DanceBuddy so we could learn to dance :)

## What it does

It breaks down poses from a dancing video into a step-by-step tutorial. Once you hit a pose, it will move on.
We break down the movements for you into key joint movements using PoseNet. With a novel cost function developed using Umeyama's research, your dance moves are graded.

## How we built it

Python, PoseNet, opencv

## Challenges we ran into

The cost function was really hard to make. Initially it was too harsh and then it was too generous.

## Accomplishments that I'm proud of

Making the best dance tutor in the world.

## What we learned

Learned a lot about machine learning

## What's next for DanceBuddy

Hopefully we can use it with a couple of our friends
## Inspiration

The hardest part of working out is getting the correct technique. One of us worked with a personal trainer to perfect his barbell squat, and it took him *months*. While personal trainers are great, they're expensive and only work for certain people (who probably aren't college students). So, the idea of an "AI personal trainer" was born. What if an AI could 'watch' somebody do an exercise and provide feedback and tips on that person's technique, with just a few clicks? We're excited just thinking about the possibilities.

## What it does

The DeepSquat app allows the user to take a front-view and side-view video of their squat using voice commands ("start" and "stop"). A very deep neural network finds keypoints on the user's body during their squat, which are used to evaluate how well the user is performing on six different attributes— whether their knees are moving too forward, whether their back is rounding, etc.— and give them an overall "squat score". In addition to the score, we provide specific feedback to allow the user to correct their biggest mistakes and improve their squat form.

## How we built it

We extracted and modified a fully-convolutional neural network from the openpose library, and used python and Apple's CoreML library to convert the network to an iOS-readable format. We then wrote Swift code to parse the network's output into heatmaps for key points on the squatter's body (right-elbow, left-knee, etc.) Using the keypoints, we developed algorithms in Swift to evaluate the quality of a squat based on six attributes. This entire process was encapsulated by the app, which we developed in Swift. Frames from both the front-view and side-view video were inputted into the network and used for squat evaluation.

## Challenges we ran into

Understanding the output of the neural network was a significant challenge. Since openpose is designed to be used with a wrapper class, there was very little information on the raw output of the network (which we'd need to use it in iOS). Using the little documentation we had, exploring the data, and making inferences, we finally did figure it out.

In addition, since the network was so large, it was not feasible to predict at a fps rate. Using the voice commands, we could extract the frame at the middle of the video (the "bottom" of the squat) and extracted keypoints from this frame and the "start" frame to evaluate squat attributes, which ended up working well.

## Accomplishments that we're proud of

We're surprised but happy at how well the Swift algorithms we created for squat evaluation worked. In order to rate "how big" the mistake a user was making is, we had to do several squats ourselves and reproduce those mistakes on video. It was a significant time investment, but we got a good sense of what "tolerance" we could provide to mistakes (for example, how far can a user's knee go beyond their ankles in the image to be a moderate error vs. a serious one).

Testing the full app for the first time and seeing it rate our squats was also pretty awesome (one of us is a lot better than the other, but we won't say who).

## What we learned

We learned how to integrate deep learning into iOS apps, as well as the most common mistakes made while doing a squat. We also learned to effectively parallelize work and how to push through bugs, frustrating as they might be.

## What's next for DeepSquat

Expand to other exercises, suggest training routines, and more— who knows, it might be a full-on personal trainer someday.
# Motion's Creation: Bringing Precision to Yoga

## Inspiration

While exploring potential ideas, MoveNet caught our attention with its capabilities for accurately tracking human movement. We recognized an opportunity: to provide wellness within reach for all through real-time feedback. Motion aspires to make wellness accessible for all. By breaking down multiple barriers, Motion allows new segmentations to receive personalized training feedback without the cost of hiring a personal trainer. Future updates to the application will ensure that anyone, irrespective of language, economic, age, or location status, can benefit.

## Implementation

We integrated a camera on our platform that effectively captures joint movements using MoveNet. To understand and analyze these movements, we utilized TensorFlow and PyTorch in our backend. Our approach involved two primary steps:

1. **Pose Prediction:** Training a machine learning model to identify the specific pose a user attempts.
2. **Pose Correction:** Training a subsequent model to detect inaccuracies in the user's pose.

If a user's pose is deemed incorrect, our system uses OpenAI's GPT API to generate unique and personalized feedback, guiding them towards the correct form.

## Challenges & Insights

Gathering diverse and representative training data posed a significant challenge. Recognizing that individuals have varying arm lengths, different distances from the camera, and diverse orientations, we aimed to make our system universally applicable. Although MoveNet expertly captures joint data in diverse scenarios, our initial model's training revealed a need for broader data. This realization led us to consider the myriad ways users might interact with our application, ensuring our model had a rich learning environment.