## Inspiration

Otter.ai, Microsoft Teams, and Dialpad. What do they have in common? They are all great options for **voice-to-text transcription**. But how about visual-to-text transcription? There are an abundance of speech-to-text transcription services, but not visual to text. This is where MotionCue comes in.

MotionCue imagines a new form of transcribing called **visual-to-text transcription** by leveraging pose detection, ML, and NLP.

We were inspired by the QHack's 2023 theme, "Designing the Digital World" and hoped to combine our interests in AI and data wrangling to ensure that information in the form of a purely visual format, especially a digital piece of media such a choreographed video, is accessible to those who are visually impaired and would appreciate text descriptions for them to understand the physical movements occuring in the video. Furthermore, text description of an image may also help sighted readers by providing them another option for processing visual information. MotionCue is bringing digital accessibility to the Performing Arts. We are bringing tools to contextualize choreography in videos. We are **digitalizing body movements** in videos, so that we can provide users with a comprehensive text guide on how to perform them.

As background, human pose estimation has been studied for over 15 years and its importance stems from the range of benefit of this technology from character animation to clinical analysis of biomechanics. [1].

Here's a brief summary on other useful applications of pose detection:

1. Translating ASL
2. Biomechanics and Medication: Human pose and movement can indicate the health status of humans
3. Sports Performance Analysis and Education: The automated extraction of poses of athletes from videos lead to greater depth of analysis of the performance of athletes and provide immediate feedback for their improvement.

## What it does

MotionCue will parse a YouTube URL and return a visual description in textual format. The application is currently focused on the genre of short dance videos. It will digitise the body movements of dance videos (such as tiktok/youtubeshorts) from YouTube. Our application will then classify the movements and dance steps and provide users with a detailed textual guide on how to perform them!

## How we built it

#### Architecture Pipeline for URL to Visual Transcription Data:

1. Use a youtube frame extractor and use it to sample images every second.
2. Process each frame to extract its pose landmark data using a pre-trained PoseDetection model (ie MoveNet, PostNet).
3. Feed each frame into the random forest model for pose detection
4. Output to frontend. Display text description in real time as the video is being played and also query OpenAI (ChatGPT) for additional information based on the labels of our pose detection model.

#### Building/training the model that generates the Visual Transcription Data:

First, we trained a quick random forest model on 2 poses by taking images manually that we recreated ourselves. Then, we found a youtube frame extractor and used it to sample images every second on tiktok videos for the popular “green green grass blue blue sky” videos. We then processed each image on the backend to extract pose landmarks. Landmarks are 2D positional data points that identify where specific body parts are on the image. We then used kmeans cluster on each image’s landmark data to identify the best key frames and corresponding pose names. After identifying the poses that we wanted to classify with our model, we trained our random forest model to classify poses. :)

## Challenges we ran into

* **Data collection:** we decided to collect, label our own dance dataset from YouTube due to the originality of our idea which consumed lots of time in the beginning and mid phase of the project. The team managed to overcome it by tight coordination and hardwork.

## Accomplishments that we're proud of

* Rethinking a new of way of transcription named "visual-to-text" transcription to help promote digital accessibility.
* Building out a fully integrated web application using Python FastAPI, React.

## What we learned

* Curating and collecting good data is hard.
* Start something really simple and build on top of that.
* Blockchain is cute.

#### Training our model

-<https://opencv.org/>
-<https://scikit-learn.org/stable/modules/ensemble.html#forest>
-<https://google.github.io/mediapipe/solutions/pose>

#### Inspecting our data

-Tensorflow pre-trained models - <https://github.com/tensorflow/tfjs-models>

#### Displaying to our ui

-<https://github.com/u-wave/react-youtube>

* <https://mui.com/material-ui/getting-started/installation/>
* tabs - <https://reactcommunity.org/react-tabs/>
#### Texts on pose detection
-<https://www.americananthro.org/ImageDescriptions?navItemNumber=25126>
-[https://www.cs.ubc.ca/~lsigal/Publications/SigalEncyclopediaCVdraft.pdf](https://www.cs.ubc.ca/%7Elsigal/Publications/SigalEncyclopediaCVdraft.pdf)

What's next for MotionCue

* training our model identifying more genres visual videos

links:
[1] [https://www.cs.ubc.ca/~lsigal/Publications/SigalEncyclopediaCVdraft.pdf](https://www.cs.ubc.ca/%7Elsigal/Publications/SigalEncyclopediaCVdraft.pdf)
[2] <https://www.sciencedirect.com/science/article/pii/S1077314221000692>
## Inspiration

Two people in our team are breakdancers and two are figure skaters. Both these communities have an incredible demand for learning new body positions and moves.

We were inspired to develop a tool to help synchronize and analyze dance and figure skating moves. SyncMe is aimed at helping dances and athletes better develop their artform through providing detailed analysis and angles of their body positions.

## Key Features

SyncMe allows users to upload a video and get feedback on how well people in the video are synchronized with each other. The backend first extracts picture frames from the video, then uses a pose-estimation neural network to find key points on people's body in each picture frame. After determining the key points, SyncMe calculates the angles of important limbs, compares people's postures with each other, and outputs a synchronizations score for each time frame. As a result, the user will be able to see when the group is not synchronized and practice accordingly.

## How we built it

For the frontend, we used HTML and javascript and built a website that allows users to browse some example images (with marked poses) and upload their own videos. For the backend, we extracted and modified a neural network from the \_ posenet \_ library to perform the pose-estimation function. Then, to enable user input of images, we created a Google Cloud storage bucket that contains images to be analyzed. To analyze the images, we calculate the angle of each limb (left arm, right arm, left torso, right torso, left leg, right leg) through the dot product of the vectors produced by the limbs. If there are multiple figures in the image, we calculate the *standard deviation* of the limb angles to see how synchronized the performers are. Thus, the lower the SS (Synchronization Score) the more in sync the performers are!

By using Google Cloud APIs, we connected the storage bucket to the front end. We further wrote a piece of frame-extraction code (in python) that extracts frames from user-uploaded images and uploads them onto Google Cloud storage bucket. After the backend calculates synchronization score, it will output the marked frames as well as the score to the webpage. The frontend and the backend are integrated by using iFrame.

## Challenges we ran into

We ran into **a lot of** \_ challenges \_.
First, no one in our team has experience with neural network, so it took us a long time to set up the environment, understand the code, and calculate the synchronization score based on the key points it outputs.
Secondly, we had difficulty integrating our frontend with the backend. We had a hard time figuring out how to use Google Cloud storage bucket and iFrame.
Thirdly, initially we could not access the Google Cloud storage bucket due to same-origin policy. We tried multiple ways to construct the Cross-origin resource sharing (CORS) request.

## Accomplishments that we're proud of

We are surprised that we used a variety of different tools in our project, and we were able to and use them as we learned. It was really great to upload our own images and see the marked output!

## What we learned

We learned how to integrate the frontend and backend of a website, how to set up a cloud storage environment, and more about how to build a website. Two people in our team has never learned javascript before (which is the main language for the neural network) and we learned along the way.

## What's next for SyncMe

Enable users to upload multiple images and compare them.
## Inspiration

Predicting the stock market is probably one of the hardest things to do in the financial world. There are so many factors that come into play when considering the rise and fall of positions, and many differing opinions on which of these factors matters the most, from technical indicators to balance sheets. It is hard to dispute, however, that one factor has gained heavy influence in the past few decades: Media. From television to the internet, the modern day media clearly has had a large impact on financial instruments and the economy in general. Taking this as inspiration, we decided to create an app that predicts future stock prices using sentiment analysis of relevant news articles.

## What it does

Mula is an app that provides an intuitive interface for users to manage their stock portfolio and predict changes that traditional methods may not catch in time. It provides a list of stocks that the user currently owns and a platform for trading them, as well as banking to store cash. Upon clicking on a specific stock, Mula provides a more detailed view of the stock’s historical prices, along with predictions of stock price changes from our algorithm coinciding with emotionally charged news. It also serves to provide a comparison between our predictions and actual price changes. Users can also put stocks on a watchlist, where they can get immediate notifications when a financial news article with significant sentiment regarding specific companies is published, as well as a recommended course of action.

## How we built it

The stack consists of a Flutter front end, which allows the app to be multi-platform, and a Python Flask backend connected to a variety of APIs. The Capital One Hackathon API, aka Nessie, is used to manage a bank account, which users of the app store their currently non-invested money. The IEX Cloud API is used to retrieve historical and current pricing data on all individual publicly traded stocks, as well as news articles written about those stocks. The Amazon Comprehend API uses machine learning to analyze those news articles and determine the emotional sentiment behind them, i.e. positive or negative. We then used a weighted mathematical model to combine that sentiment data with Growth, Integrated, and Financial Returns scores from the Goldman Sachs Marquee API to create a prediction score for how the price of an individual stock will move in the coming days.

To ensure the prediction is valid, we used the Marquee API and IEX Cloud API’s historical data and news to backtest this algorithm. We searched for news stories in the past with a significant measured sentiment, and computed a prediction of future price movement from that and the historical Growth, Integrated, and Financial Return scores. We can then look a few days ahead and see how effective this prediction was, and use that to adjust the weights of the prediction model.

## Challenges we ran into

We initially wanted to build our app using React Native, however we ran into problems trying to set up the design language we wanted to work with. We decided to switch to Flutter, which none of us had worked with before. It took some time before we got the hang of it, but ultimately we saw it as the better choice as it gave less problems. We were also working with multiple APIs, and having them be compatible with each other in the backend proved to be quite a laborious process.

## Accomplishments that we're proud of

We are very proud that we were able to successfully find a correlation between news sentiments and stock movements. This is a major finding and could set the foundation for a rise in novel trading algorithms. Additionally, we are happy that we were able to pick up Flutter relatively quickly. Now that we know how to set up apps with it, we may use it again in the future to do the same.

## What we learned

This was our first time doing a financial project for a hackathon, so we learned a lot about bank account management, risk management, technical analysis and fundamental analysis. We also learned how to combine multiple APIs together in a cohesive, valuable way. Additionally, none of us had quite done front end tasks before, so we we’re glad that we were able to learn how to use Flutter.

## What's next for Mula.

We hope to further improve on our prediction modeling algorithms and sentiment analysis with more complex neural networks/deep learning. We would also like to further iterate on the UI/UX of the app to further streamline the stock analysis and prediction process.