# Signal

We tried to build a leap motion application that allowed deaf users to use sign language to communicate.

# What does the service do?

So far, we've gotten A, B, C, and D done (with some success). We learnt quite a lot over the last 16 hours, learning how to use the Leap Motion API, learnt Javascript, and learnt how to implement a neural network to teach our program the different types of signs for each letter. We also learnt the taste of Soylent, and the various forms that Octocat comes in.

# Features

We used Brain.js to teach program all the gestures we've implemented thus far and anytime the program reads a gesture incorrectly, it will learn to correct itself over time.
## Inspiration

With apps like DuoLingo, most people have no problem spending time to learn a new language, but few people know sign language. We are a team of Western students looking to break down communication barriers by making sign language everyone's next interactive challenge.

## What it does

We use scikit-learn and Leap Motion to create an interactive way of learning and testing your sign language skills. The web app has a progression of challenges that range from identifying the alphabet, to having your signing recognized and checked.

## How to get into Signtology

1. [Clone this repository on Github](https://github.com/ivanzvonkov/hackwestern)
\*Download the prerequisites in the Requirements.txt file.
2. Open command line
3. Navigate to working directory
4. Run python app.py
5. Copy provided link from command line to browser with '\api' appended

## Challenges we ran into

The API we planned on using didn't end up working, so we had to become familiar with manipulating the frame data from the Leap Motion to define gestures.

## Accomplishments that we're proud of

Getting the leap motion to recognize the letters we sign in front of it was a big step that made the project a lot more exciting to work on.

## What we learned

It's important to stay flexible and pivot your project when your plans don't turn out to be feasible. We had to rethink our original idea on how we would store the Leap data to compare gestures.

## What's next for Signtology

We want to develop curriculum bundles that will test more complex skills (e.g spelling your name, common phrases, etc). These bundles will be available for different age groups and will lead to a proficiency in sign language.
## Inspiration

The ability to easily communicate with others is something that most of take for granted in our everyday life. However, for the millions of hearing impaired and deaf people all around the world, communicating their wants and needs is a battle they have to go through every day. The desire to make the world a more accessible place by bringing ASL to the general public in a fun and engaging manner was the motivation behind our app.

## What it does

Our app is essentially an education platform for ASL that is designed to also be fun and engaging. We provide lessons for basic ASL such as the alphabet, with plans to introduce more lessons in the future. What differentiates our app and makes it engaging, is that users can practice their ASL skills right in the app, with any new letter or word they learn, the app uses their webcam along with AI to instantly tell users when they are making the correct sign. The app also has a skills game that puts what they learnt to the test, in a time trial, that allows users to earn score for every signed letter/word. There is also a leaderboard so that users can compete globally and with friends.

## How we built it

Our app is a React app that we built with different libraries such as MUI, React Icons, Router, React-Webcam, and most importantly Fingerpose along with TensorflowJS for all our AI capabilities to recognize sign language gestures in the browser.

## Challenges we ran into

Our main struggle within this app was implementing Tensorflowjs as none of us have experience with this library prior to this event. Recognizing gestures in the browser in real time initially came with a lot of lag that led to a bad user experience, and so it took a lot of configuring and debugging in order to get a much more seamless experience.

## Accomplishments that we're proud of

As a team we were initially building another application with a similar theme that involved hardware components and we had to pivot quite late due to some unforseen complications, and so we're proud of being able to turn around with such a short amount of time and make a good product that we would be proud to show anyone. We're also proud of building a project that also has a real world usage to it that we all feel strongly about and that we think really does require a solution for.

## What we learned

Through this experience we all learned more about React as a framework, in addition to real time AI with Tensorflowjs.

## What's next for Battle Sign Language

Battle Sign Language has many more features that we would look to provide in the future, we currently have limited lessons, and our gestures are limited to the alphabet, so in the future we would increase our app to include more complex ASL such as words or sentences. We would also look forward to adding multiplayer games so that people can have fun learning and competing with friends simultaneously.