## Inspiration

We wanted this project to be able to help us share our love of music with others who may not be able to enjoy it through sound alone. Inspired by our love of music as well as technology, we hoped to make things more accessible for people with our passion for technology.

## What it does

Our ASL Music Generator takes in the user's choice of song/artist and returns the lyrics, translating it to American Sign Language

## How we built it.

We started by setting up our workspace, working on **Replit** at the start to make collaboration easier for our team, and in the later stage - GitHub and VSCode. Essentially, we prompt engineered a prompt to call using the Groq API so that we could return song lyrics with the user\_inputted variables **song\_name** and **artist**. We used javascript to connect the front-end and back-end in receiving information. We used Figma to brainstorm the format and structure of our app.

## Challenges we ran into

We ran into a lot of challenges with the setup of the project because we had spent a lot of time downloading a lot of packages as we wanted to use a large and complicated model that was able to translate text to pose/animation (American Sign Language). One of the more difficult processes was combining the front end with the back end in terms getting the song lyrics and inputting into the LLM that translates the lyrics into HamNoSys and animation/pose format. We also ran into some challenges when using Replit, it was super helpful when collaborating with a team but we also ran into a lot of efficiency issues when downloading the packages we needed to get the model to run. At the end, we had to transfer all of our code we had made on Replit into VSCode and still had to use GitHub the model we were implementing was too large.

## Accomplishments that we're proud of

We're really proud of the fact that we were able to condense such a large and complicated model to fit our project.

## What we learned

We learned that we can sleep in any position when we're tired. We also learned how to sign language "We are cooked" (No but we actually had a fun time learning a couple of sign language phrases). Additionally, we gained more experience in full-stack web development and creating a project centered around an AI model.

## What's next for ASL Lyric Generator

Our next plan is to successfully deploy the ASL Lyric Generator is to maybe implement some learning and educational features. Right now it's a lyric generator to make music more accessible to people, but we want to make this a learning experience, helping people learn sign-language through music!

`Credit for the sign.mt LLM`
[Translate Model GitHub Repo](https://github.com/sign/translate.git)
## Inspiration

We loved the idea of Image analysis and wanted to develop and app that would utilize the powerful capabilities of Microsoft's API.

## What it does

A user can upload their image on our webapp and find out which artwork from our database they look most like. Users can also add other works of art to our database to increase the variance of our Art database.

## How we built it

We used a Python-Flask framework as our webserver. mLab hosts our MongoDB and we utilize Microsoft's Face API and their SDK. We initially worked on the backend to get the program working. As we got half of the backend working, half the team started to work on the front end.

## Challenges we ran into

Coming up with idea was pretty challenging and took up a lot of time on the first day. The other challenges we were facing were utilizing an online app engine due to permission rights to write data. Given a couple more hours, we are confident that we could have this app on a live site.

## Accomplishments that we're proud of

We are extremely proud of having a working product that utilizes such a powerful tool as Microsoft Face API. We are also thrilled that we were able to use 3 sponsor's products together in one application.

## What we learned

We learned a lot about Microsoft's API's, Google's App Engine (although not utilized in this version), as well as how a user can upload a file, how to store it in a database, and how to serve it back to them.

## What's next for Doppel-Art-Ganger

We hope to deploy Doppel-Art-Ganger on a live domain as well as clean up the code, increase the Art database, and test it on our friends and family.
## Inspiration

As someone who has always wanted to speak in ASL (American Sign Language), I have always struggled with practicing my gestures, as I, unfortunately, don't know any ASL speakers to try and have a conversation with. Learning ASL is an amazing way to foster an inclusive community, for those who are hearing impaired or deaf. DuoASL is the solution for practicing ASL for those who want to verify their correctness!

## What it does

DuoASL is a learning app, where users can sign in to their respective accounts, and learn/practice their ASL gestures through a series of levels.

Each level has a *"Learn"* section, with a short video on how to do the gesture (ie 'hello', 'goodbye'), and a *"Practice"* section, where the user can use their camera to record themselves performing the gesture. This recording is sent to the backend server, where it is validated with our Action Recognition neural network to determine if you did the gesture correctly!

## How we built it

DuoASL is built up of two separate components;

**Frontend** - The Frontend was built using Next.js (React framework), Tailwind and Typescript. It handles the entire UI, as well as video collection during the *"Learn"* Section, which it uploads to the backend

**Backend** - The Backend was built using Flask, Python, Jupyter Notebook and TensorFlow. It is run as a Flask server that communicates with the front end and stores the uploaded video. Once a video has been uploaded, the server runs the Jupyter Notebook containing the Action Recognition neural network, which uses OpenCV and Tensorflow to apply the model to the video and determine the most prevalent ASL gesture. It saves this output to an array, which the Flask server reads and responds to the front end.

## Challenges we ran into

As this was our first time using a neural network and computer vision, it took a lot of trial and error to determine which actions should be detected using OpenCV, and how the landmarks from the MediaPipe Holistic (which was used to track the hands and face) should be converted into formatted data for the TensorFlow model. We, unfortunately, ran into a very specific and undocumented bug with using Python to run Jupyter Notebooks that import Tensorflow, specifically on M1 Macs. I spent a short amount of time (6 hours :) ) trying to fix it before giving up and switching the system to a different computer.

## Accomplishments that we're proud of

We are proud of how quickly we were able to get most components of the project working, especially the frontend Next.js web app and the backend Flask server. The neural network and computer vision setup was pretty quickly finished too (excluding the bugs), especially considering how for many of us this was our first time even using machine learning on a project!

## What we learned

We learned how to integrate a Next.js web app with a backend Flask server to upload video files through HTTP requests. We also learned how to use OpenCV and MediaPipe Holistic to track a person's face, hands, and pose through a camera feed. Finally, we learned how to collect videos and convert them into data to train and apply an Action Detection network built using TensorFlow

## What's next for DuoASL

We would like to:

* Integrate video feedback, that provides detailed steps on how to improve (using an LLM?)
* Add more words to our model!
* Create a practice section that lets you form sentences!
* Integrate full mobile support with a PWA!