## Background

Half **a million people** in the United States identify as deaf or hard of hearing. Accessibility has always been a huge problem, but especially nowadays as the number of people that know ASL is decreasing each year. The best time to learn a language is as a child, so we aim to target teaching ASL at the root. Our app, **HelloSign**, is an gamified, e-learning web app focused on teaching small children ASL.

## What is HelloSign?

**HelloSign** uses artificial intelligence to provide instant feedback on hand sign technique and augmented reality for an interactive and kinesthetic learning experience. Our main features include:

### Lessons & Quizzes

The student can learn through our lessons and test their skills with our quizzes. The lessons are easy to understand and the learner can learn kinesthetically with augmented reality. The quizzes use machine learning to detect the signs and provide real-time feedback.

### Badges & Prizes

We took into account that kids don't like traditional e-learning as they find it boring, so we gamified it. You can earn badges and prizes by completing lessons and quizzes

### Friends & Leaderboard

We wanted to make sure you can interact and practice with your friends through video call. We believe that socializing and friends is a part of what makes learning fun. The leaderboard is a competitive aspect to encourage children to do their very best.

### Donate Cryptocurrency

To further develop and maintain our free app, we give the option for users to donate cryptocurrency, a digital payment system that doesn't rely on banks to verify transactions, using a technology called blockchain.

## How HelloSign was built

**HelloSign** was built by a team consisting of both beginner and advanced hackers, including both designers and developers.

**HelloSign**’s design was created based on our audience, need-finding, user personas and user-flow.

**HelloSign**’s tech was made with React, Redux, Material-UI, Framer-Motion and many other technologies. Tensorflow Object Detection API and Python were used to create our own machine learning model, and then we converted our pretrained model to TensorFlow.js and hosted our model on cloud object store so that we could use it with our frontend. The backend was made with Node and Express with MongoDB as our database to store user data. Cryptocurrency transactions are made possible with Ethers and MetaMask. Finally, EchoAR was used to view 3D hand sign models in augmented reality.

## Engineering

![link](https://challengepost-s3-challengepost.netdna-ssl.com/photos/production/software_photos/001/776/667/datas/original.png)

Our tech stack.

![link](https://media.discordapp.net/attachments/920321339708031006/932092746834002010/yes.PNG?)

## UX/UI

![img](https://github.com/impedaka/codetivate/blob/main/assets/persona.PNG?raw=true)

User personas helped direct and guide us with the design of our app

### Hi-fi Prototypes

![image](https://github.com/impedaka/codetivate/blob/main/assets/Capture.PNG?raw=true)

We selected a red, green, and blue color palette, font, and developed the art style from there.

## Challenges

We used a wide variety of technologies for our frontend, our main ones being: React, Redux, CSS, JavaScript, and HTML. We also used Framer Motion for animations and Material-UI as a component library for icons and modals.

The real time object detection machine learning model was trained through transfer learning using SSD MobileNet and tensorflow object detection api from using our own dataset by labeling our images with LabelImg.As a result, it can detect hand signs in real time with your webcam and OpenCV. After making our own model, we then converted it to Tensorflow.js and hosted it through a cloud object store. From there, we could use our machine learning model with our frontend React app.

The most frustrating part would be creating our own dataset from scratch, as it was very large and time consuming. Another challenge we faced was implementing the bounding boxes for the image recognition. The feature itself was unnecessary, but it greatly improves the user interface altogether. We also struggled to provide real-time feedback/scoring, and figuring our Base64 encoding and integrating all of these components within a short period of time

## Accomplishments we are proud of

We are proud that we managed to polish and finish our app! We finished slightly earlier than the hackathon deadline, so we decided to add extra details with Material-UI icons and animations with Framer-Motion to make our user interface look more professional and organized.

## What we learned

We learned to work remotely from each other, since this was an online hackathon. We relied on discord for communication, google docs for brainstorming, figma for our design and more. The technologies and apis we used were a fun and good challenge as well.

## What’s next?

If there was more time to work on our project, we would have added a lot more features. This includes making our own 3d models to view in AR, and providing more lessons and quizzes. We would also train our machine learning model to recognize more hand signs, including hand signs that aren’t part of ASL, such as Chinese Sign Language and French Sign Language. Ultimately, we hope to see the app launch one day, we want to encourage more children to learn ASL and socialize!.
## Inspiration

Our inspiration were are friends who have Autism and also hearing the keynote speaker talk about her brother who is Autistic.

## What it does

Assist ASD is an application made for people with autism, their caretakers and behavioral therapists. Our mobile application allows people with autism to understand social signals through image/facial recognition. In addition, the care takers are able to track their receiver in order to know events like when entering and leaving home, sleeping schedules and physical activity. Care takers not only track these events, but have measures for recording notes based ABC assessment. Lastly, all information and data is then pushed to the behavioral therapist who is able to view the metrics and data analytics in order to make more informed decisions for their patients.

## How I built it

Using AGILE/SCRUM planning, we built it using technologies based on scope. After planning the project, we incorporated google's cloud service and facial recognition along with Microsoft's sentimental analytics. We incorporated Neura's API in order to get updates on the autistic person's movements such as physical activity, sleeping activity and entering or leaving designated points. We used Qualtrics in order to display data analytics of all behaviors tracked.

## Challenges I ran into

We ran into API issues with Microsoft services. Initially it was a very confusing setup because they only had code for objective C. However, after playing around with the curl and python aspect it became useful. We also had an interesting adventure of scoping our project more precisely and cutting out possible technologies that we could use.

## Accomplishments that I'm proud of

Well, we see our application having a big impact on the ASD community.

## What I learned

More about the problem space of ASD and new technologies and capabilities for applications.

## What's next for Assist ASD

We will user interview behavioral therapist to make sure we are solving and implementing the right features.
## Inspiration

In Canada alone, there are over 350,000 Canadians who are deaf and around another 3 million who are hard of hearing. However, picking up sign language for them can often be a challenge, especially for deaf children who are often born to hearing parents who often don't know sign language. The inability to communicate with and understand their peers can lead to isolation and loneliness, effectively walling them off from on of the joys of life - communication. We aim to fix this problem by not only enhancing the sign language learning process for those who are hearing impaired, but also encouraging those who are sound of hearing to pick up sign language - breaking down the communication barrier between hearing impaired people and their peers.

## What it does

Our application leverages the power of AI and neural networks to not only detect and identify sign language gestures in real-time, but also provide feedback to users so they can learn more effectively from their mistakes. We also combine this technology with engaging and interactive lessons, ensuring that learning sign language is not only effective but also enjoyable.

## How we built it

To detect hand gestures, used Python and the OpenCV library to format the images being sent through the user's webcam and MediaPipe and SciKit to detect hands gestures and predict the symbol being signed. For the Frontend, we mainly used React.js and Tailwind for the UI and CSS respectively. Finally, for the Backend, we used Express.js and Flask to handle requests from the React application and Python machine learning model respectively.

## Challenges we ran into

Training the model was a big problem as we spent a lot of time near the start trying to find a pretrained model. However, all of the pretrained models we found had very little documentation, so we weren't able to find out how to use them. We only resorted to building and training our own model very late into the hackathon, giving us very little time to make sure it meshed well with the rest of our project. We spent a lot of time dealing with React's async functions and also had a lot of trouble deploying our application.

## Accomplishments that we're proud of

We are proud of being able to accomplish what we've accomplished given the short time frame and our smaller group size.

## What we learned

To not get stuck trying to fix a single stupid bug for hours and instead move on.

## What's next for Silyntax

We aim to allow Silyntax to not only be able to recognize gestures through singular frames, but also through chaining together multiple frames into larger movements, allowing the detection of more complex gestures. We also aim to implement more game modes, such as a mode where players are given a sequence of letters/words and have to compete with one another to see who signs the sequence the fastest (kind of like Typeracer), and also a maze game mode where the player has to sign different words to move around and navigate through the maze.