## Inspiration

We love traveling, but it's a great pity to travel somewhere, take a picture of it, then forget about it. Booking a tour guide to learn about local sites is too expensive, and tourist apps are a great hassle. We want to build a complete tourist companion app that is both informative and useful.

## What it does

Once you take a picture, it automatically uploads this picture into an online album for easy management (think Google Photos). This album is updated live and interactive! Crucially, we run Google Cloud's vision analytics engine to determine what tourist attraction it is, then fetch information about it online and play it back in audio to the user.

## How we built it

The user side is built with Raspberry Pi, and once it takes a picture (controlled by a click button) that picture is sent to a server. The server runs Google Cloud Vision on the image, labels the image with a landmark name, then puts the image in a folder that is automatically synchronized with the web album that we created. If the picture does not contain a landmark, then we would use another algorithm (using pertained weights from YOLO <https://github.com/pjreddie/darknet>) to detect objects in this image.

Any information that needs to be returned to the user is put through a text-to-speech engine, which converts it into an audio file, then sent back to Raspberry Pi for playing.

## Challenges we ran into

Raspberry Pi refuses to connect to U of T internet, and we spent a long time figuring out a solution. In the end, we decided to connect the Raspberry Pi to a smartphone's hot spot, which solved the issue.

It's very difficult to send files seamlessly between Raspberry Pi and our computer. We attempted many methods and settled on using SSH, because it is not limited to local connections only (unlike Samba). This means that our program can truly work anywhere in the world.

## Accomplishments that we're proud of

It all works! Better than we thought.

## What we learned

Raspberry Pi 4, Google Cloud, OpenCV, javascript, React, IoT, SSH, Text to Speech, Python

## What's next for iStick

Optimize the time required to obtain a result: currently a bottleneck is the image-taking process, and we hope to speed this up. We can also add a screen so the user can see what the camera is capturing.
## Inspiration

When looking at the themes from the Make-a-thon, one specifically stood out to us: accessibility. We thought about common disabilities, and one that we see on a regular basis is people who are visually impaired. We thought about how people who are visually impaired navigate around the world, and we realized there isn't a good solution besides holding your phone out that allows them to get around the world. We decided we would create a device that uses Google Maps API to read directions and sense the world around it to be able to help people who are blind navigate the world without running into things.

## What it does

Based on the user's desired destination, the program reads from Google API the checkpoints needed to cross in our path and audibly directs the user on how far they are from it. Their location is also repeatedly gathered through Google API to determine their longitude and latitude. Once the user reaches
 the nearest checkpoint, they will be directed to the next checkpoint until they reach their destination.

## How we built it

Under a local hotspot host, we connected a phone and Google API to a Raspberry Pi 4. The phone would update the Raspberry Pi with our current location and Google API to determine the necessary checkpoints to get there. With all of the data being compiled on the microcontroller, it is then connected to a speaker through a Stereo Audio Amplifier Module (powered by an external power supply), which amplifies the audio sent out into the Raspberry Pi's audio jack. With all that, the directions conveyed to the user can be heard clearly.

## Challenges we ran into

Some of the challenges we faced were getting the stereo speaker to work and indicating to the user the distance from their next checkpoint, frequently within the range of the local network.

## Accomplishments that we're proud of

We were proud to have the user's current position updated according to the movement of the phone connected to the local network and be able to update the user's distance from a checkpoint in real time.

## What we learned

We learned to set up and work with a Raspberry Pi 4 through SSH.
We also learned how to use text-to-speech for the microcontroller using Python and how we can implement it in a practical application.
Finally, we were

## What's next for GPS Tracker for the Visually Impaired

During the hackathon, we were unable to implement the camera sensing the world around it to give the user live directions on what the world looks like in front of them and if they are going to run into anything or not. The next steps would include a depth camera implementation as well as an OpenCV object detection model to be able to sense the distance of things in front of you
## Inspiration

There were two primary sources of inspiration. The first one was a paper published by University of Oxford researchers, who proposed a state of the art deep learning pipeline to extract spoken language from video. The paper can be found [here](http://www.robots.ox.ac.uk/%7Evgg/publications/2018/Afouras18b/afouras18b.pdf). The repo for the model used as a base template can be found [here](https://github.com/afourast/deep_lip_reading).

The second source of inspiration is an existing product on the market, [Focals by North](https://www.bynorth.com/). Focals are smart glasses that aim to put the important parts of your life right in front of you through a projected heads up display. We thought it would be a great idea to build onto a platform like this through adding a camera and using artificial intelligence to gain valuable insights about what you see, which in our case, is deciphering speech from visual input. This has applications in aiding individuals who are deaf or hard-of-hearing, noisy environments where automatic speech recognition is difficult, and in conjunction with speech recognition for ultra-accurate, real-time transcripts.

## What it does

The user presses a button on the side of the glasses, which begins recording, and upon pressing the button again, recording ends. The camera is connected to a raspberry pi, which is a web enabled device. The raspberry pi uploads the recording to google cloud, and submits a post to a web server along with the file name uploaded. The web server downloads the video from google cloud, runs facial detection through a haar cascade classifier, and feeds that into a transformer network which transcribes the video. Upon finished, a front-end web application is notified through socket communication, and this results in the front-end streaming the video from google cloud as well as displaying the transcription output from the back-end server.

## How we built it

The hardware platform is a raspberry pi zero interfaced with a pi camera. A python script is run on the raspberry pi to listen for GPIO, record video, upload to google cloud, and post to the back-end server. The back-end server is implemented using Flask, a web framework in Python. The back-end server runs the processing pipeline, which utilizes TensorFlow and OpenCV. The front-end is implemented using React in JavaScript.

## Challenges we ran into

* TensorFlow proved to be difficult to integrate with the back-end server due to dependency and driver compatibility issues, forcing us to run it on CPU only, which does not yield maximum performance
* It was difficult to establish a network connection on the Raspberry Pi, which we worked around through USB-tethering with a mobile device

## Accomplishments that we're proud of

* Establishing a multi-step pipeline that features hardware, cloud storage, a back-end server, and a front-end web application
* Design of the glasses prototype

## What we learned

* How to setup a back-end web server using Flask
* How to facilitate socket communication between Flask and React
* How to setup a web server through local host tunneling using ngrok
* How to convert a video into a text prediction through 3D spatio-temporal convolutions and transformer networks
* How to interface with Google Cloud for data storage between various components such as hardware, back-end, and front-end

## What's next for Synviz

* With stronger on-board battery, 5G network connection, and a computationally stronger compute server, we believe it will be possible to achieve near real-time transcription from a video feed that can be implemented on an existing platform like North's Focals to deliver a promising business appeal