## Inspiration

People with hearing impairment are safe and capable drivers, although there remains some environmental cues which they cannot detect while on the road. We developed a tool that creates a safer driving experience by alerting hearing impaired drivers of other incoming vehicles.

## What it does

Three microphones are set up around a vehicle so that it samples sound from the environment in 360 degrees. If our machine-learning SVM algorithm detects an ambulance or honking car in the nearby area, it localizes the object based on the difference in amplitude of sound detected by each microphone and the time of delay between each microphone.

## How we built

We connected three microphones to an Arduino Uno for sound detection. The amplitude of sound and the time of
sound detection was read from the Arduino and channeled into a raspberry-pi for further processing. In Python, we used supervised learning with a support vector machine to detect whether a nearby sound was the siren of an ambulance or the persistent honk of a nearby car. Taking into account which microphone was persistently detecting the loudest sound, and the position of the microphone in relation to the driver, a dynamic GUI displays an arrow that tracks the position of the incoming vehicle to alert the driver.

## Difficulties encountered

Adjusting the sensitivity of the microphones so that they were providing rich enough data for training our machine learning algorithm. Interfacing the Arduino Uno with the Raspberry-Pi. Controlling our data flow so that our tool could support alerting from real-time data.

## What we're proud of

Setting up the hardware so that we could sensitively detect sounds in the environment, feeding that data into a raspberry-pi, and performing processing and supervised machine-learning in real-time.

## What we learned

We all started the project with zero experience in using Raspberry Pis or Arduinos so there was quite a learning curve for all of us. We also had never worked with triangulation, python GUIs or classifying sound data. We learned quite a bit about each topic through the course of the weekend.

## Next steps

Integrating more microphones for more sensitive and accurate 360 degree sound detection. Implementing microphones into vehicles in an effective but non-intrusive way. Training our algorithm on more environmental threats so that we may alert drivers.
# HearU

![HearU](https://challengepost-s3-challengepost.netdna-ssl.com/photos/production/software_photos/001/401/027/datas/gallery.jpg)

## Inspiration

Speech is one of the fundamental ways for us to communicate with each other, but many are left out of this channel of communication. Around half a billion people around the world suffer from hearing loss, this is something we wanted to address and help remedy.

Without the ability to hear, many are locked out of professional opportunities to advance their careers especially for jobs requiring leadership and teamwork. Even worse, many have trouble creating fulfilling relationships with others due to this communication hurdle.

Our aim is to make this an issue of the past and allow those living with hearing loss to be able to do everything they wish to do easily.
HearU is the first step in achieving this dream.
![Why it matters](https://challengepost-s3-challengepost.netdna-ssl.com/photos/production/software_photos/001/402/222/datas/gallery.jpg)

## What it does

HearU is an inexpensive and easy-to-use device designed to help those with hearing loss understand the world around them by giving them subtitles for the world.
The device attaches directly to the user’s arm and automatically transcribes any speech heard by it to the user onto the attached screen. The device then adds the transcription on a new line to make sure it is distinct for the user.
A vibration motor is also added to alert the user when a new message is added. The device also has a built-in hazard detection to determine if there are hazards in the area such as fire alarms. HearU also allows the user to store conversations that took place locally on the device for privacy.

Most Importantly, HearU allows the user to interact with ease with others around them.

## How we built it

HearU is based on a Raspberry Pi 4, and uses a display, microphone and a servo motor to operate and interface with the user. A Python script analyzes the data from the microphone, and updates the google cloud API, recording the real-time speech and then displaying the information on the attached screen. Additionally, it uses the pyaudio module to measure the decibels of the sounds surrounding the device, and it is graphically presented to the user. The frontend is then rendered with a python library called pyQT. The entire device is enclosed in a custom-built 3d printed housing unit that is attached to the user’s arm.
This project required a large number of different parts that were not standard as we were trying to minimize size as much as possible but ended up integrating beautifully together.

![hardware](https://challengepost-s3-challengepost.netdna-ssl.com/photos/production/software_photos/001/402/221/datas/gallery.jpg)

## Challenges we ran into

The biggest challenge that we ran into was hardware availability. I needed to scavenge for parts between multiple different sites, which made connecting “interesting” to say the least since we had to use 4 HDMI connectors to get the right angles we needed. Finding a screen that was affordable, that would arrive on time and the right size for the project was not easy.

Trying to fit all the components into something that would fit on an arm was extremely difficult as well and we had to compromise in certain areas such as having the power supply in the pocket for weight and size constraints. We started using a raspberry pi zero w at first but found it was too underpowered for what we wanted to do so we had to switch to the pi4. This meant we had to redesign the enclosure.

We were not able to get adequate time to build the extra features we wanted such as automatic translation since we were waiting on the delivery of the parts, as well as spending a lot of time troubleshooting the pi. We even had the power cut out for one of our developers who had all the frontend code locally saved!

In the end, we are happy with what we were able to create in the time frame given.

## Accomplishments that we're proud of

I’m proud of getting the whole hack to work at all! Like I mentioned above, the hardware was an issue to get integrated and we were under a major time crunch. Plus working with a team remotely for a hardware project was difficult and our goal was very ambitious.
We are very proud that we were able to achieve it!

## What we learned

We learned how to create dense electronic devices that manage space effectively and we now have a better understanding of the iterative engineering design process as we went through three iterations of the project. We also learned how to use google cloud as it was the first time any of us have used it. Furthermore, we learned how to create a good-looking UI using just python and pyqt. Lastly, we learned how to manipulate audio and various audio fingerprinting algorithms to match audio to sounds in the environment.

## What's next for HearU

There are many things that we can do to improve HearU.

We would like to unlock the potential for even more communication by adding automatic language translation to break down all borders.

We can also miniaturize the hardware immensely by using more custom-built electronics, to allow HearU to be as cumbersome and as easy to use as possible.

We would work on Integrating a camera to track users’ hands so that we can convert sign language into speech so that those with hearing impairments can easily communicate with others, even if they don’t know sign language.

![Future Steps](https://challengepost-s3-challengepost.netdna-ssl.com/photos/production/software_photos/001/402/223/datas/gallery.jpg)
## Inspiration

In online documentaries, we saw visually impaired individuals and their vision consisted of small apertures. We wanted to develop a product that would act as a remedy for this issue.

## What it does

When a button is pressed, a picture is taken of the user's current view. This picture is then analyzed using OCR (Optical Character Recognition) and the text is extracted from the image. The text is then converted to speech for the user to listen to.

## How we built it

We used a push button connected to the GPIO pins on the Qualcomm DragonBoard 410c. The input is taken from the button and initiates a python script that connects to the Azure Computer Vision API. The resulting text is sent to the Azure Speech API.

## Challenges we ran into

Coming up with an idea that we were all interested in, incorporated a good amount of hardware, and met the themes of the makeathon was extremely difficult. We attempted to use Speech Diarization initially but realized the technology is not refined enough for our idea. We then modified our idea and wanted to use a hotkey detection model but had a lot of difficulty configuring it. In the end, we decided to use a pushbutton instead for simplicity in favour of both the user and us, the developers.

## Accomplishments that we're proud of

This is our very first Makeathon and we were proud of accomplishing the challenge of developing a hardware project (using components we were completely unfamiliar with) within 24 hours. We also ended up with a fully function project.

## What we learned

We learned how to operate and program a DragonBoard, as well as connect various APIs together.

## What's next for Aperture

We want to implement hot-key detection instead of the push button to eliminate the need of tactile input altogether.