## Inspiration

2020 has been a difficult year for all of us. It’s been a year full of bad news, boredom, isolation and a disconnect from friends and the community. The pandemic has forced us to spend most of the year in isolation and for the most part has negatively impacted both mental and physical wellbeing. We seem to have forgotten what day of the week it is and every day feels repetitive and mundane.

## What it does

ProMotion is here to change that. We have a platform that allows you to use machine learning and augmented reality to connect with your community and make mundane tasks fun. We have created an IOS application that gives you instant feedback and helps you better yourself by correcting your form and tracking your progress in many sports such as adjusting your jump shot form in basketball and spike form in volleyball.

## How we built it

This is done through bleeding edge tech and the latest machine learning technologies available from Apple including ML kit and reality kit.

At a high level, we begin by training a machine learning model on specific action sequences. For example, a jump shot in basketball. From there we train and learn what the ideal movements for this action are. For a jump shot, we may want to train on highlights of NBA players. We can then record a user completing this specific action and capture key pivot points on their body such as their knees, elbows and wrists. We compare these pivot points to those of the ideal model and can compute the difference. Lower levels of difference results in higher scores which is signified by the user’s rating on the top right of the screen. The user can record themselves completing any action and has the ability to playback their video and view the change in their rating over time.

More specifically, getting into the technical details behind it, the tech stack is implemented as follows:

* MLKit along with the vision framework running PoseNET, a human body pose detector to capture 17 different body landmarks (wrists, ankles, knees, elbows, etc.) and perform the motion tracking. This happens live on device running at 60fps, an incredibly difficult feat
* This captured body landmark data is then fed to CreateML, where another neural network classifier classifies several types of actions (e.g. for volleyball, what a spike is, for basketball, what a jump shot is)
* Additional body landmark data is captured from reference ideal videos, including NBA highlights, etc. to give us ideal comparison models to evaluate the user’s poses against, giving us a comprehensive rating system
* Next, we had to map the comparison model to the user’s model when actually executing in app in both the time and spatial dimensions, as some people are taller and slightly different in other dimensions as well. This was a huge pain point, as we had to dynamically scale our ideal reference data.
* Finally, tying it all together we had to visualize everything, including the error/difference regions. This required us writing custom renderers drawing out the wireframe in a series of bezier paths, along with freeform polygons to visualize the error regions. Getting this right and visually appealing was quite a challenge.

## Key Features

### Practice Mode

It's been difficult to stay in shape mentally and physically and we all know that physical exercise alone can get mundane. Practice gets boring, there is no stimuli, no one to cheer you on and no way to track your progress. We recognize these major issues and ProMotion’s practice mode completely changes the game. In this mode, users can select from a large number of sports and practice essential skills and moves. There is an interactive UI in augmented reality which provides instant feedback about your form and your progress by comparing your moves to an ideal athlete.

### Challenge Mode

Our disconnect from our friends and community is easy to miss in these tough times. Even completing the simplest of tasks is a blast when doing it with those you care about. ProMotion’s challenge mode allows users to record their practices, share their progress and compete with friends.

See how you stack up against your buddies on the leaderboard and share the fun of physical activity with your friend group. Friend’s can easily share, post their attempts and you can replay them in augmented reality.

### Custom Challenges

As we’ve been alluding to, ProMotion isn’t only for athletes. The pandemic has effected everyone and ProMotion is here to help. We take our features one step further by allowing users to create custom motion instantly in our Create mode. Simply name your activity and submit a few video clips to train our ML action classifier.

Then instantly share your motion and let your friends hop on the trend. No action is too simple, even brushing your teeth or drinking water can be made into a fun activity. COVID has made everything mundane and ProMotion is rolling it back.

Furthermore, for those looking for more practical use of create modes, we help remote teachers and instructors evaluate and assess their students. Record complex yoga moves or workout techniques and share with your class to help them from a distance. They can practice these moves on their own time and compare themselves to your example.

## What's next for ProMotion

We plan to add even more functionality like commenting on your friends video or competing with them in real time. The sky is truly the limit with ProMotion.
## Inspiration

Our inspiration came from a shared interest in sports and the unfortunate realization that-- getting better was getting harder! Private Coaching was getting more expensive, and self-coaching was difficult and time-consuming. To learn how to shoot with proper basketball shooting form, an athlete would have to be willing to pay inflated prices for lessons or be left to record themselves, play, analyze the film, and scour the internet to get better. We realized that we needed a coach; we needed an AI-powered coach to make our jobs as athletes much more efficient. Inspired by the hit film *Coach Carter (2008)* and the Bostonian accent, we developed "Coach Cahtah"!

## What it does

Coach Cahtah takes a video input of a player shooting a basketball anywhere on the court and uses Machine Learning to track and trace the ball's path to the hoop so that the user can better understand how their shooting form affects their field goal percentage.

## How we built it

We used Python for the backend of the project and the public Machine Learning library, OpenCV, to track the path of the basketball in the video. Instead of using hours of nonexistent basketball training data, we used a color-mapping method on the basketball to isolate its motion and cut our time to production exponentially.

## Challenges we ran into

First off, we needed to realize how difficult image recognition was. We did not have enough workforce or existing data to fulfill our intended goals, so instead, we simplified our ambitions to reach an MVP. We also attempted to use the Terra API with the Apple Watch, but due to its closed-system nature, it wasn't easy to pull gyroscopic data from the device.

## Accomplishments that we're proud of

Fortunately enough, we were able to adapt very well considering the situation. Instead of using TensorFlow, we ended up using OpenCV, which was more beginner-friendly, and we were able to circumvent the hours of training data we needed to train a model by taking a shortcut through the whole process. We used color mapping to isolate the basketball to trace and view its path to the hoop instead.

## What we learned

Machine learning has INCREDIBLE opportunities to make our daily lives INCREDIBLY efficient. The problem is that it requires a strong understanding and high-quality training data. With time and effort, using this tool can create unique products.

## What's next for Coach Cahtah

Coach Cahtah's greatest asset is its expandability. We can integrate all of these programs into a single app to allow athletes to analyze their shooting on the court efficiently. We can also move into other sports like Tennis.
## Inspiration

Nowadays, large corporations are spending more and more money nowadays on digital media advertising, but their data collection tools have not been improving at the same rate. Nike spent over $3.03 billion on advertising alone in 2014, which amounted to approximately $100 per second, yet they only received a marginal increase in profits that year. This is where Scout comes in.

## What it does

Scout uses a webcam to capture facial feature data about the user. It sends this data through a facial recognition engine in Microsoft Azure's Cognitive Services to determine demographics information, such as gender and age. It also captures facial expressions throughout an Internet browsing session, say a video commercial, and applies sentiment analysis machine learning algorithms to instantaneously determine the user's emotional state at any given point during the video. This is also done through Microsoft Azure's Cognitive Services. Content publishers can then aggregate this data and analyze it later to determine which creatives were positive and which creatives generated a negative sentiment.

Scout follows an opt-in philosophy, so users must actively turn on the webcam to be a subject in Scout. We highly encourage content publishers to incentivize users to participate in Scout (something like $100/second) so that both parties can benefit from this platform.

We also take privacy very seriously! That is why photos taken through the webcam by Scout are not persisted anywhere and we do not collect any personal user information.

## How we built it

The platform is built on top of a Flask server hosted on an Ubuntu 16.04 instance in Azure's Virtual Machines service. We use nginx, uWSGI, and supervisord to run and maintain our web application. The front-end is built with Google's Materialize UI and we use Plotly for complex analytics visualization.

The facial recognition and sentiment analysis intelligence modules are from Azure's Cognitive Services suite, and we use Azure's SQL Server to persist aggregated data. We also have an Azure Chatbot Service for data analysts to quickly see insights.

## Challenges we ran into

**CORS CORS CORS!.** Cross-Origin Resource Sharing was a huge pain in the head for us. We divided the project into three main components: the Flask backend, the UI/UX visualization, and the webcam photo collection+analysis. We each developed our modules independently of each other, but when we tried to integrate them together, we ran into a huge number of CORS issues with the REST API endpoints that were on our Flask server. We were able to resolve this with a couple of extra libraries but definitely a challenge figuring out where these errors were coming from.

SSL was another issue we ran into. In 2015, Google released a new WebRTC Policy that prevented webcam's from being accessed on insecure (HTTP) sites in Chrome, with the exception of localhost. This forced us to use OpenSSL to generate self-signed certificates and reconfigure our nginx routes to serve our site over HTTPS. As one can imagine, this caused havoc for our testing suites and our original endpoints. It forced us to resift through most of the code we had already written to accommodate this change in protocol. We don't like implementing HTTPS, and neither does Flask apparently. On top of our code, we had to reconfigure the firewalls on our servers which only added more time wasted in this short hackathon.

## Accomplishments that we're proud of

We were able to multi-process our consumer application to handle the massive amount of data we were sending back to the server (2 photos taken by the webcam each second, each photo is relatively high quality and high memory).

We were also able to get our chat bot to communicate with our REST endpoints on our Flask server, so any metric in our web portal is also accessible in Messenger, Skype, Kik, or whatever messaging platform you prefer. This allows marketing analysts who are frequently on the road to easily review the emotional data on Scout's platform.

## What we learned

When you stack cups, start with a 3x3 base and stack them in inverted directions.

## What's next for Scout

You tell us! Please feel free to contact us with your ideas, questions, comments, and concerns!