## Inclusiv.ai: Empowering Accessibility

## Inspiration

In a world where technology is a cornerstone of daily life, it's crucial that digital access is equitable. However, not everyone experiences technology in the same way. We took this challenge to heart and crafted Inclusiv.ai to revolutionize accessibility, ensuring that individuals with disabilities can navigate the web with ease and confidence. Our main priority was reducing the hassle of different extensions and toggles. Inclusiv.ai has one button and one assistant—Inki.

## What it does

Inclusiv provides a simple way for those with disabilities to navigate the web. We focused on a hassle-free, intuitive approach with only one button. Simply begin a conversation with your assistant, "Inki," by clicking on the microphone button and explain whatever issues have been limiting your experience on the web. With various modes such as colorblind, screen enhancer, screen explainer and summarizer, and an ADHD/Dyslexia mode.

Inki is designed to be a dynamic tool, adapting to a variety of needs through its multiple modes:
Colorblind Mode: Tailors the web page colors, ensuring that colorblind users can differentiate between colors that are typically hard to distinguish.
Screen Enhancer: Amplifies and clarifies website content for those with visual impairments, allowing for easier reading and interaction.
Screen Explainer and Summarizer: A mode that not only explains the elements on the screen but also provides concise summaries for quick comprehension, beneficial for users with cognitive disabilities.
ADHD/Dyslexia Mode: Alters the web page layout and typography to minimize distractions and optimize for readability, assisting users with attention deficits or dyslexia.
Behind these user-facing features, Inclusiv utilizes a combination of large language models and a specific focus on Monster API. This approach has enabled us to create features that are both technically sophisticated and varied, ensuring a broad range of user needs are met.

## How we built it

Inclusiv was meticulously crafted by a synergistic team of two backend developers and one frontend designer, all united by the vision of making web navigation universally accessible. The project was born from a user-centric approach, focusing on the unique challenges faced by individuals with disabilities. Our team's design philosophy hinged on simplicity, leading to the creation of a one-button interface that calls upon Inki, an AI assistant, to activate different accessibility modes including colorblind, screen enhancer, screen explainer, and summarizer, and an ADHD/Dyslexia mode. By leveraging large language models for natural language processing and integrating Monster API's robust algorithms, Inclusiv transcends conventional assistive technologies. This blend of intuitive design and advanced AI capabilities was continuously refined through iterative user testing and feedback, ensuring that the product genuinely resonates with the needs of its users. Inclusiv’s launch is not an end, but a beginning to an ongoing journey of innovation and improvement, with a commitment to evolving and expanding its features to foster an inclusive web experience for all.

## Challenges we ran into

Where to start?
Front-end might be harder than the back end! We spent hours fiddling with the optimal UI setup, trying to figure out how best to simplify the experience for a user of any skill level. This is not as simple as it looks, and we spent two hours trying to add a power button. Additionally, we ended up having to pivot our AI model numerous times and switched our approach throughout the project.

## Accomplishments that we're proud of

We ended up changing our AI agent multiple times throughout the project to deliver the optimal project. We're really satisfied with our ability to adapt and push ourselves out of our comfort zones throughout the 36 hours. We developed numerous, technically difficult and varied features that we believe will truly aid those with any form of disability.

## What we learned

We learned tons about building a project from scratch, implementing LLMs, and creating an intuitive and appealing front-end experience.

## What's next for Inclusiv.ai

We want to launch on the Chrome web store!
## Inspiration

Over **15% of American adults**, over **37 million** people, are either **deaf** or have trouble hearing according to the National Institutes of Health. One in eight people have hearing loss in both ears, and not being able to hear or freely express your thoughts to the rest of the world can put deaf people in isolation. However, only 250 - 500 million adults in America are said to know ASL. We strongly believe that no one's disability should hold them back from expressing themself to the world, and so we decided to build Sign Sync, **an end-to-end, real-time communication app**, to **bridge the language barrier** between a **deaf** and a **non-deaf** person. Using Natural Language Processing to analyze spoken text and Computer Vision models to translate sign language to English, and vice versa, our app brings us closer to a more inclusive and understanding world.

## What it does

Our app connects a deaf person who speaks American Sign Language into their device's camera to a non-deaf person who then listens through a text-to-speech output. The non-deaf person can respond by recording their voice and having their sentences translated directly into sign language visuals for the deaf person to see and understand. After seeing the sign language visuals, the deaf person can respond to the camera to continue the conversation.

We believe real-time communication is the key to having a fluid conversation, and thus we use automatic speech-to-text and text-to-speech translations. Our app is a web app designed for desktop and mobile devices for instant communication, and we use a clean and easy-to-read interface that ensures a deaf person can follow along without missing out on any parts of the conversation in the chat box.

## How we built it

For our project, precision and user-friendliness were at the forefront of our considerations. We were determined to achieve two critical objectives:

1. Precision in Real-Time Object Detection: Our foremost goal was to develop an exceptionally accurate model capable of real-time object detection. We understood the urgency of efficient item recognition and the pivotal role it played in our image detection model.
2. Seamless Website Navigation: Equally essential was ensuring that our website offered a seamless and intuitive user experience. We prioritized designing an interface that anyone could effortlessly navigate, eliminating any potential obstacles for our users.

* Frontend Development with Vue.js: To rapidly prototype a user interface that seamlessly adapts to both desktop and mobile devices, we turned to Vue.js. Its flexibility and speed in UI development were instrumental in shaping our user experience.
* Backend Powered by Flask: For the robust foundation of our API and backend framework, Flask was our framework of choice. It provided the means to create endpoints that our frontend leverages to retrieve essential data.
* Speech-to-Text Transformation: To enable the transformation of spoken language into text, we integrated the webkitSpeechRecognition library. This technology forms the backbone of our speech recognition system, facilitating communication with our app.
* NLTK for Language Preprocessing: Recognizing that sign language possesses distinct grammar, punctuation, and syntax compared to spoken English, we turned to the NLTK library. This aided us in preprocessing spoken sentences, ensuring they were converted into a format comprehensible by sign language users.
* Translating Hand Motions to Sign Language: A pivotal aspect of our project involved translating the intricate hand and arm movements of sign language into a visual form. To accomplish this, we employed a MobileNetV2 convolutional neural network. Trained meticulously to identify individual characters using the device's camera, our model achieves an impressive accuracy rate of 97%. It proficiently classifies video stream frames into one of the 26 letters of the sign language alphabet or one of the three punctuation marks used in sign language. The result is the coherent output of multiple characters, skillfully pieced together to form complete sentences

## Challenges we ran into

Since we used multiple AI models, it was tough for us to integrate them seamlessly with our Vue frontend. Since we are also using the webcam through the website, it was a massive challenge to seamlessly use video footage, run realtime object detection and classification on it and show the results on the webpage simultaneously. We also had to find as many opensource datasets for ASL as possible, which was definitely a challenge, since with a short budget and time we could not get all the words in ASL, and thus, had to resort to spelling words out letter by letter. We also had trouble figuring out how to do real time computer vision on a stream of hand gestures of ASL.

## Accomplishments that we're proud of

We are really proud to be working on a project that can have a profound impact on the lives of deaf individuals and contribute to greater accessibility and inclusivity. Some accomplishments that we are proud of are:

* Accessibility and Inclusivity: Our app is a significant step towards improving accessibility for the deaf community.
* Innovative Technology: Developing a system that seamlessly translates sign language involves cutting-edge technologies such as computer vision, natural language processing, and speech recognition. Mastering these technologies and making them work harmoniously in our app is a major achievement.
* User-Centered Design: Crafting an app that's user-friendly and intuitive for both deaf and hearing users has been a priority.
* Speech Recognition: Our success in implementing speech recognition technology is a source of pride.
* Multiple AI Models: We also loved merging natural language processing and computer vision in the same application.

## What we learned

We learned a lot about how accessibility works for individuals that are from the deaf community. Our research led us to a lot of new information and we found ways to include that into our project. We also learned a lot about Natural Language Processing, Computer Vision, and CNN's. We learned new technologies this weekend. As a team of individuals with different skillsets, we were also able to collaborate and learn to focus on our individual strengths while working on a project.

## What's next?

We have a ton of ideas planned for Sign Sync next!

* Translate between languages other than English
* Translate between other sign languages, not just ASL
* Native mobile app with no internet access required for more seamless usage
* Usage of more sophisticated datasets that can recognize words and not just letters
* Use a video image to demonstrate the sign language component, instead of static images
## Inspiration

After growing up learning programming concepts from great tools such as Scratch, Logo and Processing, we realised the power but also the limitations of using visual feedback to make learning programming more rewarding.

Tools like these and many others, rely on visual mediums to communicate core programming ideas to those who are new to programming.

However, people with visual impairments are locked out of learning to program in this intuitive, rewarding manner.

We wanted to break down this barrier, and bring them the power of instant feedback learning techniques, vastly increasing the accessibility of programming education.

## What it does

Our project provides an environment for people with visual impairments to learn core computer science concepts.

Our project contains a fully featured online editor which uses text to speech technologies to make text editing accessible by speaking the code the user is interacting with back to them in an intuitive way.

The editor allows users to program in a novel, innovative language designed specifically to provide an excellent learning experience, with no setup required.
With this language users can use a fun, intuitive and beginner-friendly API to create all manner of sounds and music.
This language brings the instant feedback used in visual beginner languages, such as Scratch, to the domain of audio.

Users can write code in their browser, and our custom interpreter and audio engine will compile and play whatever sounds and music their hearts desire.

## How we built it

Our back-end is written in Python, it uses the TextX PEG parser generator and pydub audio manipulation library.
Our online editor is written in Angular.

## Challenges we ran into

It turns out that designing and implementing a language from scratch is a lot harder than we predicted.
We also did not expect that the audio manipulation would be as challenging as it was.
Our final major hurdle that we overcame was the streaming of the generated sound back to the users browser.

## Accomplishments that we're proud of

We are very proud of our final product, and we firmly believe that it can be a very useful accessibility tool for visually impaired learners in the future.

Our language syntax was specifically designed to be as friendly to text to speech editing as possible, minimising use of special characters, and matching English speech as closely as possible. We are extremely proud of how this turned out and how readable (literally) it is.

## What we learned

Each of our team members learned much about their respective areas of the project. Between language design, audio manipulation, and dev tool design we were forced to familiarise ourselves quickly with many areas of computer science in which we had no prior experience.
There was a lot of co-operation and pair programming so the new knowledge was well distributed among the team.

## What's next for Muse

We have great plans to continue our development of this project throughout the coming years in the hope that this project will enable as many people as possible to enter the world of computer science that otherwise may not have had the opportunity.