## 💡 INSPIRATION

We first thought of building something for the deaf & blind because one of our team member's close family relations has slowly been acquiring age-related dual vision and hearing loss.

Deafblindness has been traditionally overlooked. The combination of sensory losses means existing solutions, such as sign language for the deaf or speech transcription for the blind, cannot be used. As a result, over 160 million Deafblind people around the world are severely limited in their ability to communicate.

Current solutions require uncomfortable physical contact or the presence of a 24/7 guide-communicator, preventing independence. These barriers to being able to freely explore and interact with your world can lead to you feeling isolated and alone.

## 🎯 OUR GOAL

Our goal was to build a communication device that can help Deafblind persons understand and communicate with the world without invasive physical contact, a requirement for a volunteer communicator, or an inaccessible/impractical design.

---

# 🔭 WHAT IS PERCEIV/IO?

PERCEIV/IO is a novel hardware device powered by the latest AI advancements to help the Deafblind.

We designed PERCEIV/IO to have two main use-cases. First, used a camera to take periodic captures of the world in front of the user and use AI image-to-text technologies to translate it into braille, which is then displayed on a tactile hand-held device that we have fabricated. This can help a Deafblind person understand their surroundings, which is currently very limited in scope. Second, we designed a speech-to-braille system to help allow for communication with the Deafblind. We also incorporated sentiment analysis to effectively convey the tones of standard English phrases to make communication more robust. The specific details can be found in the next section.

We hope some of the ideas generated during this hackathon turn are able to be used in organizations such as [CNIB Deafblind Community Services](https://deafblindservices.ca/about-us) in order to help them with their goal of assisting people who need it most.

## 📣 FEATURES

* Get a description of the world in front of you translated into braille directly at your fingertips.
* Live English-to-braille translation & sentiment analysis to allow for more robust communication.
* Store a profile of the user so PERCEIV/IO can recognize common occurrences such as people and places.
* An accompanying, fully functional, universally deployed web app that allows you to view your profile and update it with any necessary info.

## 🛠️ HOW WE BUILT IT

### Hardware

We designed our hardware using four stepper motors and their drivers, 3D printed braille discs, 3D printed stepper motor housings, a Raspberry Pi 3B, and custom 3D printed housings. Each stepper motor controls an octagonal braille disc. Placing two of these side-by-side creates a full braille cell, with different stepper motor rotations producing different braille characters. We designed the Raspberry Pi code on Python and developed our own algorithim for converting text directly into stepper motor coordinates (position to display the correct braille character).

### Software

We used image recognition AI based on **Groq** using their Llava v1.5 image-to-text endpoint, and that text is then put through **Groq**'s Llama 3.1 70b endpoint to condense it down into just two or three words. We used Google Speech-to-Text and the Facebook RoBERTa base model for simple sentiment analysis with a **Genesys**-based sentiment analysis feature currently being deployed. We developed our own code to convert into Grade II Unified English Braille Code.

On the backend, our database is based on **Convex** along with our accompanying app. We also built a prototype, web-hosted app on the Streamlit platform integrated with hugging face ML models loaded through the transformers pipeline.

We facilitated wireless communication between the apps and the hardware through Github by running functions to update a file and check for updates on a file. We are also working on a **Genesys** based AI chat bot, the responses of which would be translated into braille and communicated to the user.

## 🌎 ENVIRONMENTALLY CONSCIOUS

As users of AI during this boom of innovation over the last couple years, we strive to be as environmentally conscious as possible and be aware of the power we are consuming whenever we prompt an LLM. Over the course of a large model's first one million messages, it has an estimated power cost of [55.1Mwh](https://adasci.org/how-much-energy-do-llms-consume-unveiling-the-power-behind-ai/). That's enough power to power roughly 120 American homes for a year!

Thankfully, with **Groq**, not only do we have access to lightning-fast response speeds, but due to their LPUs, SRAM, and sequential processing we are using less power than if we were to use the traditional NVIDIA counterpart. Conserving energy is very important to us, and we are thankful for Groq to providing a solution for us.

## 🧗‍♀️ CHALLENGES WE'VE OVERCOME

Hackathons are about nothing if not overcoming challenges. Whenever a piece of hardware broke, software wouldn't cooperate, or motors wouldn't connect, we would work as a team to solve the issue and push on through. It ended up being a very rewarding experience for all of us.

One major challenge that's still fresh in my mind is our issues with the Raspberry Pi. We ended up going through *three* Pi's before we realized that none of their SD cards had any OS on them! Eventually, with the help of the amazing mentors here at Hack The North (thank you again Moody!), we we're able to get the SD card flashed and were able to start uploading our code. Thankfully, due to our adoption of SOLID principles and our policy of encapsulation, we were able to implement the code we has worked on in the meantime with ease.

Our hardware issues didn't end there! After careful testing of almost every different software & hardware component to determine why our code was not working, we found that 2 stepper motor drivers were faulty. With the help of mentors, however, we got this resolved and we learned the painstakingly careful process of hardware verification!

Another problem was more to do with logic- turning an image into text is easy. Turning that text into braille, and then turning that braille into signals from a Raspberry Pi into 4 proprietary stepper motors that control half of a braille character each is whole other. Luckily with some wisdom from our team members with past experience with these kinds of electronics combined with some logic learned in hardware classes, we as a team were able to come up with the implementation to make it work.

## 🏆 WHAT WE'RE PROUD OF

We are most proud of coming up with a novel idea that uses novel technology to help people in need. I think it is everyone's desire to develop something that is actually useful to other humans, especially while utilizing the latest technologies. We think that technology's greatest advantages are the advantages it is able to give to others in order to promote diversity, equity and inclusion.

### Some Specifics

The device our team members' relative would have to use costs upwards of $3000 USD ([link](https://store.humanware.com/hus/brailliant-bi-40x-braille-display.html)). This is prohibitively expensive, especially given that many of the world's Deafblind come from underprivileged nations. The prototyping costs for our hardware were free, thanks to Hack the North. However, we estimate manufacturing costs to be under $100 given the low-cost hardware, making this system far more affordable.

Second, our system is novel in its comprehensive approach. Current solutions (such as the $3000 product) require a connection to an external computer to function. This means they cannot be used portably, thereby preventing a Deafblind person from being able to explore. Moreover, they do not integrate any of the image-to-braille or speech & sentiment-to-braille technologies, severely limiting their scope. We believe our project will substantially improve the status quo by allowing for these features. Our device is portable, can be used without additional peripherals, and integrates all necessary conversion technology.

Lastly, our prototype has two fully-functioning braille cells that *follow the distance and format requirements of Grade II Braille*. This means that our hardware is *not just a proof of concept* for a braille device, but could actually be implemented directly in Deafblind communities. Moreover, it is likely to be extremely efficient as the stepper motor design does not require individual motors for each braille pin, but rather just one motor for a whole column.

**Perhaps most importantly, though...**

We are so proud of the fact that we came up with and created a working prototype for our idea within only 32 hours(!!!). For many of us, this was our first time working with hardware within such a short time frame, so learning the ins and outs of it enough to make a product is a huge accomplishment.

## 💡 WHAT DID WE LEARN?

Over the course of this hackathon, we learned so much about the Deafblind community by reading online testimonials of those who have been diagnosed. It allowed us insight into a corner of the world that we had otherwise not known much about prior. Through developing our product while keeping these testimonials in mind, we also realized the difference between developing a product in order to beat others in a competition and developing a product because we believe this could actually be useful to real people one day.

The many sponsors here at Hack The North were also very valuable in teaching us about their product and how we can implement them into our product to improve it's functionality and efficiency. For example, Groq were very helpful in describing exactly why utilizing their API was more energy efficient than the big guys. We were super eager to learn about new technologies such as Symphonic labs, as we realized their use-case as an AI that can read lips was very applicable to our device.

## 🔜 THE FUTURE OF PERCEIV/IO

We have thought long and hard about the future of PERCEIV/IO. We've already come up with a laundry list of ideas that we want to implement sometime in the future in order for this product to achieve it's full potential. The number one thing is to shrink the size of the actual hardware. Raspberry Pi's are extremely useful for prototyping- but are a bit too bulky for use in a commercial product. We want to achieve a size similar to a smart watch with similar processing power.

We also want to relocate the camera and have it wirelessly communicate with the computer. One idea was to embed the camera into a pair of sunglasses in a similar style to [Spectacles](https://www.spectacles.com/ca-en/) by Snap Inc.

And finally, we are waiting for the day where AI technology and hardware reaches a point where we can run all models locally, without the need for a massive power bank or network connection. That would truly untether the user from any external requirements for a real feeling of freedom.

---

### 🪿 Thank you to all who showed interested in PERCEIV/IO, and a huge congratulations to *all* hackers who submitted in time. **We did it!**
## Inspiration

Over **15% of American adults**, over **37 million** people, are either **deaf** or have trouble hearing according to the National Institutes of Health. One in eight people have hearing loss in both ears, and not being able to hear or freely express your thoughts to the rest of the world can put deaf people in isolation. However, only 250 - 500 million adults in America are said to know ASL. We strongly believe that no one's disability should hold them back from expressing themself to the world, and so we decided to build Sign Sync, **an end-to-end, real-time communication app**, to **bridge the language barrier** between a **deaf** and a **non-deaf** person. Using Natural Language Processing to analyze spoken text and Computer Vision models to translate sign language to English, and vice versa, our app brings us closer to a more inclusive and understanding world.

## What it does

Our app connects a deaf person who speaks American Sign Language into their device's camera to a non-deaf person who then listens through a text-to-speech output. The non-deaf person can respond by recording their voice and having their sentences translated directly into sign language visuals for the deaf person to see and understand. After seeing the sign language visuals, the deaf person can respond to the camera to continue the conversation.

We believe real-time communication is the key to having a fluid conversation, and thus we use automatic speech-to-text and text-to-speech translations. Our app is a web app designed for desktop and mobile devices for instant communication, and we use a clean and easy-to-read interface that ensures a deaf person can follow along without missing out on any parts of the conversation in the chat box.

## How we built it

For our project, precision and user-friendliness were at the forefront of our considerations. We were determined to achieve two critical objectives:

1. Precision in Real-Time Object Detection: Our foremost goal was to develop an exceptionally accurate model capable of real-time object detection. We understood the urgency of efficient item recognition and the pivotal role it played in our image detection model.
2. Seamless Website Navigation: Equally essential was ensuring that our website offered a seamless and intuitive user experience. We prioritized designing an interface that anyone could effortlessly navigate, eliminating any potential obstacles for our users.

* Frontend Development with Vue.js: To rapidly prototype a user interface that seamlessly adapts to both desktop and mobile devices, we turned to Vue.js. Its flexibility and speed in UI development were instrumental in shaping our user experience.
* Backend Powered by Flask: For the robust foundation of our API and backend framework, Flask was our framework of choice. It provided the means to create endpoints that our frontend leverages to retrieve essential data.
* Speech-to-Text Transformation: To enable the transformation of spoken language into text, we integrated the webkitSpeechRecognition library. This technology forms the backbone of our speech recognition system, facilitating communication with our app.
* NLTK for Language Preprocessing: Recognizing that sign language possesses distinct grammar, punctuation, and syntax compared to spoken English, we turned to the NLTK library. This aided us in preprocessing spoken sentences, ensuring they were converted into a format comprehensible by sign language users.
* Translating Hand Motions to Sign Language: A pivotal aspect of our project involved translating the intricate hand and arm movements of sign language into a visual form. To accomplish this, we employed a MobileNetV2 convolutional neural network. Trained meticulously to identify individual characters using the device's camera, our model achieves an impressive accuracy rate of 97%. It proficiently classifies video stream frames into one of the 26 letters of the sign language alphabet or one of the three punctuation marks used in sign language. The result is the coherent output of multiple characters, skillfully pieced together to form complete sentences

## Challenges we ran into

Since we used multiple AI models, it was tough for us to integrate them seamlessly with our Vue frontend. Since we are also using the webcam through the website, it was a massive challenge to seamlessly use video footage, run realtime object detection and classification on it and show the results on the webpage simultaneously. We also had to find as many opensource datasets for ASL as possible, which was definitely a challenge, since with a short budget and time we could not get all the words in ASL, and thus, had to resort to spelling words out letter by letter. We also had trouble figuring out how to do real time computer vision on a stream of hand gestures of ASL.

## Accomplishments that we're proud of

We are really proud to be working on a project that can have a profound impact on the lives of deaf individuals and contribute to greater accessibility and inclusivity. Some accomplishments that we are proud of are:

* Accessibility and Inclusivity: Our app is a significant step towards improving accessibility for the deaf community.
* Innovative Technology: Developing a system that seamlessly translates sign language involves cutting-edge technologies such as computer vision, natural language processing, and speech recognition. Mastering these technologies and making them work harmoniously in our app is a major achievement.
* User-Centered Design: Crafting an app that's user-friendly and intuitive for both deaf and hearing users has been a priority.
* Speech Recognition: Our success in implementing speech recognition technology is a source of pride.
* Multiple AI Models: We also loved merging natural language processing and computer vision in the same application.

## What we learned

We learned a lot about how accessibility works for individuals that are from the deaf community. Our research led us to a lot of new information and we found ways to include that into our project. We also learned a lot about Natural Language Processing, Computer Vision, and CNN's. We learned new technologies this weekend. As a team of individuals with different skillsets, we were also able to collaborate and learn to focus on our individual strengths while working on a project.

## What's next?

We have a ton of ideas planned for Sign Sync next!

* Translate between languages other than English
* Translate between other sign languages, not just ASL
* Native mobile app with no internet access required for more seamless usage
* Usage of more sophisticated datasets that can recognize words and not just letters
* Use a video image to demonstrate the sign language component, instead of static images
## welcome to Catmosphere!

we wanted to make a game with (1) cats and (2) cool art. inspired by the many "cozy indie" games on steam and on social media, we got working on a game where the cat has to avoid all the obstacles as it attempts to go into outer space.

**what it does**: use the WASD keys to navigate our cat around the enemies. enter the five levels of the atmosphere and enjoy the art and music while you're at it!

**what's next for Catmosphere**: adding more levels, a restart button, & a new soundtrack and artwork