## Inspiration

There are thousands of people worldwide who suffer from conditions that make it difficult for them to both understand speech and also speak for themselves. According to the Journal of Deaf Studies and Deaf Education, the loss of a personal form of expression (through speech) has the capability to impact their (affected individuals') internal stress and lead to detachment. One of our main goals in this project was to solve this problem, by developing a tool that would a step forward in the effort to make it seamless for everyone to communicate. By exclusively utilizing mouth movements to predict speech, we can introduce a unique modality for communication.

While developing this tool, we also realized how helpful it would be to ourselves in daily usage, as well. In areas of commotion, and while hands are busy, the ability to simply use natural lip-reading in front of a camera to transcribe text would make it much easier to communicate.

## What it does

**The Speakinto.space website-based hack has two functions: first and foremost, it is able to 'lip-read' a stream from the user (discarding audio) and transcribe to text; and secondly, it is capable of mimicking one's speech patterns to generate accurate vocal recordings of user-inputted text with very little latency.**

## How we built it

We have a Flask server running on an AWS server (Thanks for the free credit, AWS!), which is connected to a Machine Learning model running on the server, with a frontend made with HTML and MaterializeCSS. This was trained to transcribe people mouthing words, using the millions of words in LRW and LSR datasets (from the BBC and TED). This algorithm's integration is the centerpiece of our hack. We then used the HTML MediaRecorder to take 8-second clips of video to initially implement the video-to-mouthing-words function on the website, using a direct application of the machine learning model.

We then later added an encoder model, to translate audio into an embedding containing vocal information, and then a decoder, to convert the embeddings to speech. To convert the text in the first function to speech output, we use the Google Text-to-Speech API, and this would be the main point of future development of the technology, in having noiseless calls.

## Challenges we ran into

The machine learning model was quite difficult to create, and required a large amount of testing (and caffeine) to finally result in a model that was fairly accurate for visual analysis (72%). The process of preprocessing the data, and formatting such a large amount of data to train the algorithm was the area which took the most time, but it was extremely rewarding when we finally saw our model begin to train.

## Accomplishments that we're proud of

Our final product is much more than any of us expected, especially the ability to which it seemed like it was an impossibility when we first started. We are very proud of the optimizations that were necessary to run the webpage fast enough to be viable in an actual use scenario.

## What we learned

The development of such a wide array of computing concepts, from web development, to statistical analysis, to the development and optimization of ML models, was an amazing learning experience over the last two days. We all learned so much from each other, as each one of us brought special expertise to our team.

## What's next for speaking.space

As a standalone site, it has its use cases, but the use cases are limited due to the requirement to navigate to the page. The next steps are to integrate it in with other services, such as Facebook Messenger or Google Keyboard, to make it available when it is needed just as conveniently as its inspiration.
## Inspiration

When the first experimental COVID-19 vaccine became available in China, hundreds of people started queuing outside hospitals, waiting to get that vaccine. Imagine this on a planetary scale when the whole everybody has to be vaccinated all around the world. There's a big chance while queuing they can spread the virus to people around them or maybe get infected because they cannot perform social distancing at all. We sure don't want that to happen.

The other big issue is that there are lots of conspiracy theories, rumors, stigma, and other forms of disinformation simultaneously spread across our social media about COVID-19 and it's vaccine. This misinformation creates frustrations for users many asking, we really don't know which one is right? Which one is wrong?

## What it does

Immunize is a mobile app that can save your life and save your time. The goal is to make the distribution of mass-vaccination become more effective, faster, and less crowded. With this app, you can book your vaccine appointment based on your own preference. So the user can easily choose the hospital based on the nearest location and easily schedule an appointment based on their availability.

In addition, based on the research we found that most of Covid-19 vaccines requires 2 doses given in 3 weeks apart to achieve that high effectiveness. And there's a big probability that people can forget to return for a follow-up shot. We can minimize that probability. This app will automatically schedule the patient for the 2nd vaccination so there is a less likelihood of user error. The reminder system (as notification feature) that will remind them in their phone when they have appointment that day.

## How we built it

We built the prototype using flutter as our client to support mobile. We integrated radar.io for hospital search. For facial recognition we used GCP and SMS reminders we used twilio. The mobile client connected to firebase: using firebase for auth, firebase storage for avatars and firestore for user metadata storage. The second backend host used datastax.

## Challenges we ran into

Working with an international team was very challenging with team members 12+ hours apart. All of us were learning something new whether it was flutter, facial recognition or experimenting with new APIs. Flutter APIs were very experimental, the camera API had to be rolled back two major version which occurred in less than 2 months to find a viable working version compatible with online tutorials

## Accomplishments that we're proud of

The features:

1. **QR Code Feature** for storing all personal data + health condition, so user don't need to wait for a long queue of administrative things.
2. **Digital Registration Form** checking if user is qualified of COVID-19 vaccine and which vaccine suits best.
3. **Facial Recognition** due to potential fraud in people who are not eligible for vaccination attempting to get limited supplies of vaccine, we implemented facial recognition to confirm the user for the appointment is the same one that showed up.
4. **Scheduling Feature** based on date, vaccine availability, and the nearby hospital.
5. **Appointment History** to track all data of patients, this data can be used for better efficiency of mass-vaccination in the future.
6. **Immunize Passport** for vaccine & get access to public spaces. This will create domino effect for people to get vaccine as soon as possible so that they can get access.
7. **Notification** to remind the patients every time they have appointment/ any important news via SMS and push notifications
8. **Vaccine Articles** - to ensure the user can get the accurate information from a verified source.
9. **Emergency Button** - In case there are side effects after vaccination.
10. **Closest Hospitals/Pharmacies** - based on a user's location, users can get details about the closest hospitals through Radar.io Search API.

## What we learned

We researched and learned a lot about the facts of COVID-19 Vaccine; Some coronavirus vaccines may work better in certain populations than others. And there may be one vaccine that seems to work better in the elderly than in younger populations. Alternatively, one may work better in children than it works in the elderly. Research suggests, the coronavirus vaccine will likely require 2 shots to be effective in which taken 21 days apart for Pfizer's vaccine and 28 days apart for Moderna's remedy.

## What's next for Immunize

Final step is to propose this solution to our government. We really hope this app could be implemented in real life and be a solution for people to get COVID-19 vaccine effectively, efficiently, and safely. Polish up our mobile app and build out an informational web app and a mobile app for hospital staff to scan QR codes and verify patient faces (currently they have to use the same app as the client)
## Inspiration

In Canada alone, there are over 350,000 Canadians who are deaf and around another 3 million who are hard of hearing. However, picking up sign language for them can often be a challenge, especially for deaf children who are often born to hearing parents who often don't know sign language. The inability to communicate with and understand their peers can lead to isolation and loneliness, effectively walling them off from on of the joys of life - communication. We aim to fix this problem by not only enhancing the sign language learning process for those who are hearing impaired, but also encouraging those who are sound of hearing to pick up sign language - breaking down the communication barrier between hearing impaired people and their peers.

## What it does

Our application leverages the power of AI and neural networks to not only detect and identify sign language gestures in real-time, but also provide feedback to users so they can learn more effectively from their mistakes. We also combine this technology with engaging and interactive lessons, ensuring that learning sign language is not only effective but also enjoyable.

## How we built it

To detect hand gestures, used Python and the OpenCV library to format the images being sent through the user's webcam and MediaPipe and SciKit to detect hands gestures and predict the symbol being signed. For the Frontend, we mainly used React.js and Tailwind for the UI and CSS respectively. Finally, for the Backend, we used Express.js and Flask to handle requests from the React application and Python machine learning model respectively.

## Challenges we ran into

Training the model was a big problem as we spent a lot of time near the start trying to find a pretrained model. However, all of the pretrained models we found had very little documentation, so we weren't able to find out how to use them. We only resorted to building and training our own model very late into the hackathon, giving us very little time to make sure it meshed well with the rest of our project. We spent a lot of time dealing with React's async functions and also had a lot of trouble deploying our application.

## Accomplishments that we're proud of

We are proud of being able to accomplish what we've accomplished given the short time frame and our smaller group size.

## What we learned

To not get stuck trying to fix a single stupid bug for hours and instead move on.

## What's next for Silyntax

We aim to allow Silyntax to not only be able to recognize gestures through singular frames, but also through chaining together multiple frames into larger movements, allowing the detection of more complex gestures. We also aim to implement more game modes, such as a mode where players are given a sequence of letters/words and have to compete with one another to see who signs the sequence the fastest (kind of like Typeracer), and also a maze game mode where the player has to sign different words to move around and navigate through the maze.