## Background

Half **a million people** in the United States identify as deaf or hard of hearing. Accessibility has always been a huge problem, but especially nowadays as the number of people that know ASL is decreasing each year. The best time to learn a language is as a child, so we aim to target teaching ASL at the root. Our app, **HelloSign**, is an gamified, e-learning web app focused on teaching small children ASL.

## What is HelloSign?

**HelloSign** uses artificial intelligence to provide instant feedback on hand sign technique and augmented reality for an interactive and kinesthetic learning experience. Our main features include:

### Lessons & Quizzes

The student can learn through our lessons and test their skills with our quizzes. The lessons are easy to understand and the learner can learn kinesthetically with augmented reality. The quizzes use machine learning to detect the signs and provide real-time feedback.

### Badges & Prizes

We took into account that kids don't like traditional e-learning as they find it boring, so we gamified it. You can earn badges and prizes by completing lessons and quizzes

### Friends & Leaderboard

We wanted to make sure you can interact and practice with your friends through video call. We believe that socializing and friends is a part of what makes learning fun. The leaderboard is a competitive aspect to encourage children to do their very best.

### Donate Cryptocurrency

To further develop and maintain our free app, we give the option for users to donate cryptocurrency, a digital payment system that doesn't rely on banks to verify transactions, using a technology called blockchain.

## How HelloSign was built

**HelloSign** was built by a team consisting of both beginner and advanced hackers, including both designers and developers.

**HelloSign**’s design was created based on our audience, need-finding, user personas and user-flow.

**HelloSign**’s tech was made with React, Redux, Material-UI, Framer-Motion and many other technologies. Tensorflow Object Detection API and Python were used to create our own machine learning model, and then we converted our pretrained model to TensorFlow.js and hosted our model on cloud object store so that we could use it with our frontend. The backend was made with Node and Express with MongoDB as our database to store user data. Cryptocurrency transactions are made possible with Ethers and MetaMask. Finally, EchoAR was used to view 3D hand sign models in augmented reality.

## Engineering

![link](https://challengepost-s3-challengepost.netdna-ssl.com/photos/production/software_photos/001/776/667/datas/original.png)

Our tech stack.

![link](https://media.discordapp.net/attachments/920321339708031006/932092746834002010/yes.PNG?)

## UX/UI

![img](https://github.com/impedaka/codetivate/blob/main/assets/persona.PNG?raw=true)

User personas helped direct and guide us with the design of our app

### Hi-fi Prototypes

![image](https://github.com/impedaka/codetivate/blob/main/assets/Capture.PNG?raw=true)

We selected a red, green, and blue color palette, font, and developed the art style from there.

## Challenges

We used a wide variety of technologies for our frontend, our main ones being: React, Redux, CSS, JavaScript, and HTML. We also used Framer Motion for animations and Material-UI as a component library for icons and modals.

The real time object detection machine learning model was trained through transfer learning using SSD MobileNet and tensorflow object detection api from using our own dataset by labeling our images with LabelImg.As a result, it can detect hand signs in real time with your webcam and OpenCV. After making our own model, we then converted it to Tensorflow.js and hosted it through a cloud object store. From there, we could use our machine learning model with our frontend React app.

The most frustrating part would be creating our own dataset from scratch, as it was very large and time consuming. Another challenge we faced was implementing the bounding boxes for the image recognition. The feature itself was unnecessary, but it greatly improves the user interface altogether. We also struggled to provide real-time feedback/scoring, and figuring our Base64 encoding and integrating all of these components within a short period of time

## Accomplishments we are proud of

We are proud that we managed to polish and finish our app! We finished slightly earlier than the hackathon deadline, so we decided to add extra details with Material-UI icons and animations with Framer-Motion to make our user interface look more professional and organized.

## What we learned

We learned to work remotely from each other, since this was an online hackathon. We relied on discord for communication, google docs for brainstorming, figma for our design and more. The technologies and apis we used were a fun and good challenge as well.

## What’s next?

If there was more time to work on our project, we would have added a lot more features. This includes making our own 3d models to view in AR, and providing more lessons and quizzes. We would also train our machine learning model to recognize more hand signs, including hand signs that aren’t part of ASL, such as Chinese Sign Language and French Sign Language. Ultimately, we hope to see the app launch one day, we want to encourage more children to learn ASL and socialize!.
## Story behind the project

When COVID-19 cancelled in-person classes and learning was forced to transition online, students were faced with many learning challenges including decreased support from teachers, lack of daily structure, and general boredom. With these issues on the forefront of remote education, we created worldclass to help make the transition to remote learning for young kids easy, fun, and accessible.

## What is 🌎worldclass

Worldclass is an online educational web platform that strives to provide an engaging and seamless learning experience for young students. By gamifying the eLearning experience, students can complete lessons to earn accessories for their “World”, allowing them to add fixtures like playgrounds, sandboxes, swings, and other things you can find in a school yard.

Teachers can input their questions into a database which will get output through a series of mini-quizzes. The students will then take these mini-quizzes to not only test their knowledge but to also earn prizes for their person world. We wanted to promote continuous learning so if a student wants to improve on their result from a quiz they have the option to re-do them.

## How we built it

We built our database using PostgreSQL. This includes tables for each of the students, teachers, lessons, questions, and answers. We then connected our database to Autocode. There, we created all of our endpoints in NodeJS. We built all of our front-end on ReactJS which leverages the different endpoints created in Autocode. These components were styled using HTML and CSS based on our designs from Figma.

## Challenges we ran into

Tech Challenges:

* learning how to make endpoints on Autocode
* connecting Autocode with PostgreSQL

Design Challenges:

* due to time constraints, not being able to implement all designs envisioned in development

## Accomplishments that we're proud of

One of the aspects we are most proud of in our product is the interface/visual design and user experience.

We’re also happy to have not only been introduced to Autocode in this hackathon, but also incorporated it into our hack. For some members, it was their first time coding in React and was a fun learning experience.

Lastly, the way our team worked together and helped each other out in times of need is something we're all proud of.

## What's next for 🌎 worldclass

We see potential in 🌎worldclass to be on the forefront of ed-tech for younger audiences.

Our hope for 🌎worldclass is to expand the platform for a wider audience. The timeframe of the hackathon had us prioritize specific goals in our project, but our vision for 🌎worldclass is to expand the interactivity on the students side for the lessons. We also hope to include more student-to-student interaction. Our research showed that student-to-student interaction is the thing kids missed the most from remote learning, so giving the students the ability to visit their classmates’s Worlds is something we hope to include.

Accessibility is of huge importance to us which we've wanted to implement since the early stages of our project. In the future we would love to incorporate customizable features/filters for the application to aid with accessibility, such as in font choice, size, colour use, and use of audio/text-to-speech.

Github Repo: <https://github.com/viet-quocnguyen/ht6ix>
## Inspiration

Fashion has always been a world that seemed far away from tech. We want to bridge this gap with "StyleList", which understands your fashion within a few swipes and makes personalized suggestions for your daily outfits. When you and I visit the Nordstorm website, we see the exact same product page. But we could have completely different styles and preferences. With Machine Intelligence, StyleList makes it convenient for people to figure out what they want to wear (you simply swipe!) and it also allows people to discover a trend that they favor!

## What it does

With StyleList, you don’t have to scroll through hundreds of images and filters and search on so many different websites to compare the clothes. Rather, you can enjoy a personalized shopping experience with a simple movement from your fingertip (a swipe!). StyleList shows you a few clothing items at a time. Like it? Swipe left. No? Swipe right! StyleList will learn your style and show you similar clothes to the ones you favored so you won't need to waste your time filtering clothes. If you find something you love and want to own, just click “Buy” and you’ll have access to the purchase page.

## How I built it

We use a web scrapper to get the clothing items information from Nordstrom.ca and then feed these data into our backend. Our backend is a Machine Learning model trained on the bank of keywords and it provides next items after a swipe based on the cosine similarities between the next items and the liked items. The interaction with the clothing items and the swipes is on our React frontend.

## Accomplishments that I'm proud of

Good teamwork! Connecting the backend, frontend and database took us more time than we expected but now we have a full stack project completed. (starting from scratch 36 hours ago!)

## What's next for StyleList

In the next steps, we want to help people who wonders "what should I wear today" in the morning with a simple one click page, where they fill in the weather and plan for the day then StyleList will provide a suggested outfit from head to toe!