## **Horus - A Voice-Activated Visual Aid**

## Inspiration

In a world increasingly dependent on visual information, we identified a significant gap in assistive technologies for the visually impaired, especially concerning the day-to-day task of identifying objects and reading labels. Initially, our vision was limited to aiding the visually impaired in reading medication labels effectively. However, as we delved deeper into the capabilities of AI models, our horizon expanded, nurturing the ambition to develop a tool that could potentially redefine how visually impaired individuals interact with their surroundings, broadening the scope to encompass a myriad of objects and reading materials.

## What it does

Horus stands as a beacon of independence and empowerment for the visually impaired community. This innovative voice-activated tool utilizes state-of-the-art AI technology to facilitate effortless object identification and label reading through simple voice commands such as, "read" and "scan". Whether it's recognizing everyday items or reading labels on medicine bottles, Horus strives to make the world more accessible, one voice command at a time.

## How we built it

We designed Horus leveraging Google's Vision API to integrate potent AI models capable of detecting objects and extracting text from images for seamless label reading. Our system utilizes OpenCV to manage and process the video feed, while Google's Web Speech API stands as the backbone for voice command recognition, ensuring a fluid user interaction with the application.

## Challenges we ran into

One of the many challenges we encountered was the initial plan to deploy Horus as a mobile application - a plan hindered by our limited experience in mobile development. Nonetheless, innovation prevailed as we ingeniously repurposed our smartphones into IP cameras, facilitating live feed capture through the web. This adaptive approach allowed us to bypass our initial hurdles, integrating the video and audio feeds into our application for real-time detection and interaction.

## Accomplishments we're proud of

Despite encountering obstacles in the development journey, we are immensely proud of our resilience and innovation. Our team demonstrated a remarkable ability to pivot and adapt, converting a setback into a stepping stone for progress. The successful integration of the IP camera as a workaround exhibits our resourcefulness and determination to bring Horus to life, setting a firm foundation for future enhancements and iterations of the project. Furthermore, the cross-referencing with a dictionary to ensure that only accurate and relevant words are returned back to users is another feature that we are excited to continue developing.

## What we learned

Through the journey of developing Horus, we have garnered invaluable insights into the boundless potential of AI technology, especially its power to bridge gaps and foster inclusivity. Furthermore, we learned the importance of flexibility and innovative thinking in the face of challenges, fostering a spirit of resilience and collaborative problem-solving that we believe will be instrumental in our future endeavors.

## What's next for Horus

Looking ahead, we envision Horus evolving into a standalone mobile application, offering users an enhanced level of independence and functionality. We are eager to explore the integration of a 'Find Feature' that would allow users to locate specific items through vocal commands, further enriching the user experience and promoting autonomy. With more time, we would aim to add a flash feature to enable users to use Horus in low-light settings, as well as noise filtering to improve use in loud environments. Additionally, we strive to develop the text-to-speech to be more natural and human-like, simulating the cadence of natural language. We also aim to associate certain packaging abbreviations with English words, such as reading 'g' as 'grams' from nutritional food labels).

Our journey with Horus is just beginning, and we are excited to continue pushing the boundaries of what's possible, one step at a time.
## Inspiration

As we sat down to brainstorm ideas for our next project, we were struck by a common thread that connected all of us. Each one of us had a family member who suffered from some form of visual impairment. It was a heart-wrenching reminder of the challenges that these individuals face on a daily basis. We shared stories of our loved ones struggling to read books, watch movies, or even navigate through everyday tasks. It was a deeply emotional conversation that left us feeling both empathetic and determined to make a difference. According to the World Health Organization, approximately 2.2 billion people worldwide have a vision impairment or blindness, with the majority of cases occurring in low and middle-income countries. The impact of visual impairment is far-reaching and significantly affects various daily activities such as reading, recognizing faces, navigating unfamiliar environments, and accessing information on digital platforms. This problem is valid, and it needs to be addressed to enhance the quality of life of those affected. We are passionate about developing a solution that will make a meaningful difference in the lives of those affected by visual impairment. Our project is inspired by personal experiences and fueled by a desire to make a real-world impact. We believe that everyone deserves equal access to information and the ability to participate fully in daily life. By addressing the challenges of visual impairment, we hope to create a more inclusive world for all.

## What it does

The product aims to bridge the gap for individuals with limited vision to experience the world around them. It helps individuals with visual impairments to perform various daily activities that are otherwise challenging, such as reading, recognizing faces, and navigating unfamiliar environments. It also assists in accessing information on digital platforms. The product can be particularly helpful for those who face barriers in accessing healthcare services due to their visual impairments. It can aid in reading prescription labels, understanding medical instructions, and navigating healthcare facilities, especially for older individuals who are aging.

## How we built it

In our project, we leverage cutting-edge computer vision techniques to interpret the surrounding environment of individuals with visual impairments. By utilizing advanced algorithms and neural networks, we process real-time visual data captured by a camera, enabling us to identify and analyze objects, obstacles, and spatial cues in the user's surroundings. We integrate state-of-the-art language models and natural language generation powered by Wisp AI software to bridge the gap between the interpreted world and the user. This allows us to generate detailed and contextually relevant descriptions of the environment in real time, providing visually impaired individuals with comprehensive auditory feedback about their surroundings. Additionally, our solution extends beyond descriptive capabilities to enhance accessibility in public transportation. By leveraging the interpreted environmental data, we develop guidance systems that assist users in navigating through streets and accessing transportation hubs safely and independently. For efficient and scalable deployment of our model, we utilize Intel's AI environment, leveraging its robust infrastructure and resources to host and optimize our machine learning algorithms. Our system architecture is implemented on a Raspberry Pi embedded platform, equipped with a high-resolution camera for real-time visualization and data capture. This combination of hardware and software components enables seamless integration and efficient visual information processing, empowering visually impaired individuals with enhanced mobility and independence in their daily lives.

## Challenges we ran into

As beginners in machine learning, we faced the tough challenge of setting up a machine learning model on a Raspberry Pi and connecting it to a camera, which was quite difficult to learn. Moreover, we had to figure out a way to train our model not only to understand text but also to recognize public transportation and calculate the distance to a bus entrance, which was quite a task. Adding our Intel-AI environment to the project made things even more complicated. Additionally, finding an affordable solution that could be easily accessible to people all around the world was a significant obstacle that we had to overcome.

## Accomplishments that we're proud of

Through this process of building a hardware product from scratch and learning how to use raspberry pi with computer vision, we not only gained technical knowledge but also learned how to work as a team. There were challenges and obstacles along the way, but we figured it out by collaborating, communicating, and leveraging each other's strengths. It was a great learning experience, and we are proud of what we have achieved together.

## What we learned

We learned about LLM, real-time text analysis, real-time text comprehension, and implementation of text-to-speech.

## What's next for True-Sight

With the growing potential of Artificial Intelligence, our idea of True-Sight is expanding to include not only text recognition but also the ability to detect surroundings, which could greatly benefit public transportation users who rely on finding stops and navigating their way onto the correct buses/trains. After further development, True-Sight could potentially allow users to locate their desired stop and use environment detection to guide them towards the door with specific step-by-step instructions. In addition, we aim to make True-Sight accessible to children who are visually impaired, so they can have an immersive learning experience. Adding sensors and custom software will also allow for a more personalized and relatable experience with the AI assistant.
## Inspiration

Millions of people around the world are either blind, or partially sighted. For those who's vision is impaired, but not lost, there are tools that can help them see better. By increasing contrast and detecting lines in an image, some people might be able to see clearer.

## What it does

We developed an AR headset that processes the view in front of it and displays a high contrast image. It also has the capability to recognize certain images and can bring their existence to the attention of the wearer (one example we used was looking for crosswalk signs) with an outline and a vocal alert.

## How we built it

OpenCV was used to process the image stream from a webcam mounted on the VR headset, the image is processed with a Canny edge detector to find edges and contours. Further a BFMatcher is used to find objects that resemble a given image file, which is highlighted if found.

## Challenges we ran into

We originally hoped to use an oculus rift, but we were not able to drive the headset with the available hardware. We opted to use an Adafruit display mounted inside a Samsung VR headset instead, and it worked quite well!

## Accomplishments that we're proud of

Our development platform was based on macOS 10.12, Python 3.5 and OpenCV 3.1.0, and OpenCV would not cooperate with our OS. We spent many hours compiling and configuring our environment until it finally worked. This was no small feat. We were also able to create a smooth interface using multiprocessing, which operated much better than we expected.

## What we learned

Without the proper environment, your code is useless.

## What's next for EyeSee

Existing solutions exist, and are better suited for general use. However a DIY solution is endlessly customizable, we this project inspires other developers to create projects that help other people.

## Links

Feel free to read more about visual impairment, and how to help;
<https://w3c.github.io/low-vision-a11y-tf/requirements.html>