## Inspiration

One In every 250 people suffer from cerebral palsy, where the affected person cannot move a limb properly, And thus require constant care throughout their lifetimes. To ease their way of living, we have made this project, 'para-pal'.

The inspiration for this idea was blended with a number of research papers and a project called Pupil which used permutations to make communication possible with eye movements.

## What it does

![Main](https://media.discordapp.net/attachments/828211308305448983/828261879326572544/iris_seg.png?width=819&height=355)

**"What if Eyes can Speak? Yesss - you heard it right!"**
Para-pal is a novel idea that tracks patterns in the eye movement of the patient and then converts into actual speech. We use the state-of-the-art iris recognition (dlib) to accurately track the eye movements to figure out the the pattern. Our solution is sustainable and very cheap to build and setup. Uses QR codes to connect the caretaker and the patient's app.

We enable paralyzed patients to **navigate across the screen using their eye movements**. They can select an action by placing the cursor for more than 3 seconds or alternatively, they can **blink three times to select the particular action**. A help request is immediately sent to the mobile application of the care taker as a **push notification**

## How we built it

We've embraced flutter in our frontend to make the UI - simple, intuitive with modularity and customisabilty. The image processing and live-feed detection are done on a separate child python process. The iris-recognition at it's core uses dlib and pipe the output to opencv.
We've developed a desktop-app (which is cross-platform with a rpi3 as well)for the patient and a mobile app for the caretaker.

We also tried running our desktop application on Raspberry Pi using an old laptop screen. In the future, we wish to make a dedicated hardware which can be cost-efficient for patients with paralysis.

![hardware](https://media.discordapp.net/attachments/828211308305448983/828263070228676638/20210404_191100.jpg?width=542&height=406)

![hardware2](https://media.discordapp.net/attachments/828211308305448983/828263051420762182/20210404_191120.jpg?width=542&height=406)

## Challenges we ran into

Building up the dlib took a significant amount of time, because there were no binaries/wheels and we had to build from source. Integrating features to enable connectivity and sessions between the caretaker's mobile and the desktop app was hard. Fine tuning some parameters of the ML model, preprocessing and cleaning the input was a real challenge.

Since we were from a different time zone, it was challenging to stay awake throughout the 36 hours and make this project!

## Accomplishments that we're proud of

* An actual working application in such a short time span.
* Integrating additional hardware of a tablet for better camera accuracy.
* Decoding the input feed with a very good accuracy.
* Making a successful submission for HackPrinceton.
* Team work :)

## What we learned

* It is always better to use a pre-trained model than making one yourself, because of the significant accuracy difference.
* QR scanning is complex and is harder to integrate in flutter than how it looks on the outside.
* Rather than over-engineering a flutter component, search if a library exists that does exactly what is needed.

## What's next for Para Pal - What if your eyes can speak?

* More easier prefix-less code patterns for the patient using an algorithm like huffman coding.
* More advanced controls using ML that tracks and learns the patient's regular inputs to the app.
* Better analytics to the care-taker.
* More UI colored themes.
# Inspiration and Product

There's a certain feeling we all have when we're lost. It's a combination of apprehension and curiosity – and it usually drives us to explore and learn more about what we see. It happens to be the case that there's a [huge disconnect](http://www.purdue.edu/discoverypark/vaccine/assets/pdfs/publications/pdf/Storylines%20-%20Visual%20Exploration%20and.pdf) between that which we see around us and that which we know: the building in front of us might look like an historic and famous structure, but we might not be able to understand its significance until we read about it in a book, at which time we lose the ability to visually experience that which we're in front of.

Insight gives you actionable information about your surroundings in a visual format that allows you to immerse yourself in your surroundings: whether that's exploring them, or finding your way through them. The app puts the true directions of obstacles around you where you can see them, and shows you descriptions of them as you turn your phone around. Need directions to one of them? Get them without leaving the app. Insight also supports deeper exploration of what's around you: everything from restaurant ratings to the history of the buildings you're near.

## Features

* View places around you heads-up on your phone - as you rotate, your field of vision changes in real time.
* Facebook Integration: trying to find a meeting or party? Call your Facebook events into Insight to get your bearings.
* Directions, wherever, whenever: surveying the area, and find where you want to be? Touch and get instructions instantly.
* Filter events based on your location. Want a tour of Yale? Touch to filter only Yale buildings, and learn about the history and culture. Want to get a bite to eat? Change to a restaurants view. Want both? You get the idea.
* Slow day? Change your radius to a short distance to filter out locations. Feeling adventurous? Change your field of vision the other way.
* Want get the word out on where you are? Automatically check-in with Facebook at any of the locations you see around you, without leaving the app.

# Engineering

## High-Level Tech Stack

* NodeJS powers a RESTful API powered by Microsoft Azure.
* The API server takes advantage of a wealth of Azure's computational resources:
	+ A Windows Server 2012 R2 Instance, and an Ubuntu 14.04 Trusty instance, each of which handle different batches of geospatial calculations
	+ Azure internal load balancers
	+ Azure CDN for asset pipelining
	+ Azure automation accounts for version control
* The Bing Maps API suite, which offers powerful geospatial analysis tools:
	+ RESTful services such as the Bing Spatial Data Service
	+ Bing Maps' Spatial Query API
	+ Bing Maps' AJAX control, externally through direction and waypoint services
* iOS objective-C clients interact with the server RESTfully and display results as parsed

## Application Flow

iOS handles the entirety of the user interaction layer and authentication layer for user input. Users open the app, and, if logging in with Facebook or Office 365, proceed through the standard OAuth flow, all on-phone. Users can also opt to skip the authentication process with either provider (in which case they forfeit the option to integrate Facebook events or Office365 calendar events into their views).

After sign in (assuming the user grants permission for use of these resources), and upon startup of the camera, requests are sent with the user's current location to a central server on an Ubuntu box on Azure. The server parses that location data, and initiates a multithread Node process via Windows 2012 R2 instances. These processes do the following, and more:

* Geospatial radial search schemes with data from Bing
* Location detail API calls from Bing Spatial Query APIs
* Review data about relevant places from a slew of APIs

After the data is all present on the server, it's combined and analyzed, also on R2 instances, via the following:

* Haversine calculations for distance measurements, in accordance with radial searches
* Heading data (to make client side parsing feasible)
* Condensation and dynamic merging - asynchronous cross-checking from the collection of data which events are closest

Ubuntu brokers and manages the data, sends it back to the client, and prepares for and handles future requests.

## Other Notes

* The most intense calculations involved the application of the [Haversine formulae](https://en.wikipedia.org/wiki/Haversine_formula), i.e. for two points on a sphere, the central angle between them can be described as:

![Haversine 1](https://upload.wikimedia.org/math/1/5/a/15ab0df72b9175347e2d1efb6d1053e8.png)

and the distance as:

![Haversine 2](https://upload.wikimedia.org/math/0/5/5/055b634f6fe6c8d370c9fa48613dd7f9.png)

(the result of which is non-standard/non-Euclidian due to the Earth's curvature). The results of these formulae translate into the placement of locations on the viewing device.

These calculations are handled by the Windows R2 instance, essentially running as a computation engine. All communications are RESTful between all internal server instances.

## Challenges We Ran Into

* *iOS and rotation*: there are a number of limitations in iOS that prevent interaction with the camera in landscape mode (which, given the need for users to see a wide field of view). For one thing, the requisite data registers aren't even accessible via daemons when the phone is in landscape mode. This was the root of the vast majority of our problems in our iOS, since we were unable to use any inherited or pre-made views (we couldn't rotate them) - we had to build all of our views from scratch.
* *Azure deployment specifics with Windows R2*: running a pure calculation engine (written primarily in C# with ASP.NET network interfacing components) was tricky at times to set up and get logging data for.
* *Simultaneous and asynchronous analysis*: Simultaneously parsing asynchronously-arriving data with uniform Node threads presented challenges. Our solution was ultimately a recursive one that involved checking the status of other resources upon reaching the base case, then using that knowledge to better sort data as the bottoming-out step bubbled up.
* *Deprecations in Facebook's Graph APIs*: we needed to use the Facebook Graph APIs to query specific Facebook events for their locations: a feature only available in a slightly older version of the API. We thus had to use that version, concurrently with the newer version (which also had unique location-related features we relied on), creating some degree of confusion and required care.

## A few of Our Favorite Code Snippets

A few gems from our codebase:

```
var deprecatedFQLQuery = '...

```

*The story*: in order to extract geolocation data from events vis-a-vis the Facebook Graph API, we were forced to use a deprecated API version for that specific query, which proved challenging in how we versioned our interactions with the Facebook API.

```
addYaleBuildings(placeDetails, function(bulldogArray) {
    addGoogleRadarSearch(bulldogArray, function(luxEtVeritas) {
        ...

```

*The story*: dealing with quite a lot of Yale API data meant we needed to be creative with our naming...

```
// R is the earth's radius in meters
var a = R * 2 * Math.atan2(Math.sqrt((Math.sin((Math.PI / 180)(latitude2 - latitude1) / 2) * Math.sin((Math.PI / 180)(latitude2 - latitude1) / 2) + Math.cos((Math.PI / 180)(latitude1)) * Math.cos((Math.PI / 180)(latitude2)) * Math.sin((Math.PI / 180)(longitude2 - longitude1) / 2) * Math.sin((Math.PI / 180)(longitude2 - longitude1) / 2))), Math.sqrt(1 - (Math.sin((Math.PI / 180)(latitude2 - latitude1) / 2) * Math.sin((Math.PI / 180)(latitude2 - latitude1) / 2) + Math.cos((Math.PI / 180)(latitude1)) * Math.cos((Math.PI / 180)(latitude2)) * Math.sin((Math.PI / 180)(longitude2 - longitude1) / 2) * Math.sin((Math.PI / 180)(longitude2 - longitude1) / 2);)));

```

*The story*: while it was shortly after changed and condensed once we noticed it's proliferation, our implementation of the Haversine formula became cumbersome very quickly. Degree/radian mismatches between APIs didn't make things any easier.
## Inspiration

1.3 billion People have some sort of vision impairment. They face difficulties in simple day to day task like reading, recognizing faces, objects, etc. Despite the huge number surprisingly there are only a few devices in the market to aid them, which can be hard on the pocket (5000$ - 10000$!!). These devices essentially just magnify the images and only help those with mild to moderate impairment. There is no product in circulation for those who are completely blind.

## What it does

The Third Eye brings a plethora of features at just 5% the cost. We set our minds to come up with a device that provides much more than just a sense of sight and most importantly is affordable to all. We see this product as an edge cutting technology for futuristic development of Assistive technologies.

## Feature List

**Guidance** - ***Uses haptic feedback to navigate the user to the room they choose avoiding all obstacles***. In fact, it's a soothing robotic head massage guiding you through the obstacles around you. Believe me, you're going to love it.

**Home Automation** - ***Provides full control over all house appliances***. With our device, you can just call out **Alexa** or even using the mobile app and tell it to switch off those appliances directly from the bed now.

**Face Recognition** - Recognize friends (even their emotions (P.S: Thanks to **Cloud Vision's** accurate facial recognitions!). Found someone new? Don't worry, on your command, we register their face in our database to ensure the next meeting is no more anonymous and awkward!

**Event Description** - ***Describes the activity taking place***. A group of people waving somewhere and you still not sure what's going on next to you? Fear not, we have made this device very much alive as this specific feature gives speech feedback describing the scenic beauty around you. [Thanks to **Microsoft Azure API**]

**Read Up** - You don't need to spend some extra bucks for blind based products like braille devices. Whether it be general printed text or a handwritten note. With the help of **Google Cloud Vision**, we got you covered from both ends. **Read up** not only decodes the text from the image but using **Google text to speech**, we also convert the decoded data into a speech so that the blind person won't face any difficulty reading any kind of books or notes they want.

**Object Locator** - Okay, so whether we are blind or not, we all have this bad habit of misplacing things. Even with the two eyes, sometimes it's too much pain to find the misplaced things in our rooms. And so, we have added the feature of locating most generic objects within the camera frame with its approximate location. You can either ask for a specific object which you're looking for or just get the feedback of all the objects **Google Cloud Vision** has found for you.

**Text-a-Friend** - In the world full of virtuality and social media, we can be pushed back if we don't have access to the fully connected online world. Typing could be difficult at times if you have vision issues and so using **Twilio API** now you can easily send text messages to saved contacts.

**SOS** - Okay, so I am in an emergency, but I can't find and trigger the SOS feature!? Again, thanks to the **Twilio** messaging and phone call services, with the help of our image and sensor data, now any blind person can ***Quickly intimate the authorities of the emergency along with their GPS location***. (This includes auto-detection of hazards too)

**EZ Shoppe** - It's not an easy job for a blind person to access ATMs or perform monetary transactions independently. And so, taking this into consideration, with the help of superbly designed **Capital One Hackathon API**, we have created a **server-based blockchain** transaction system which adds ease to your shopping without being worried about anything. Currently, the server integrated module supports **customer addition, account addition, person to person transactions, merchant transactions, balance check and info, withdrawals and secure payment to vendors**. No need of worrying about individual items, just with one QR scan, your entire shopping list is generated along with the vendor information and the total billing amount.

**What's up Doc** - Monitoring heart pulse rate and using online datasets, we devised a machine learning algorithm and classified labels which tells about the person's health. These labels include: "Athletic", "Excellent", 'Good", "Above Average", "Average", "Below Average" and "Poor". The function takes age, heart rate, and gender as an argument and performs the computation to provide you with the best current condition of your heart pulse rate.

\*All features above can be triggered from Phone via voice, Alexa echo dot and even the wearable itself.
\*\*Output information is relayed via headphones and Alexa.

## How we built it

Retrofit Devices (NodeMCU) fit behind switchboards and allow them to be controlled remotely.

The **RSSI guidance uses Wi-Fi signal intensity** to triangulate its position. Ultrasonic sensor and camera detects obstacles (**OpenCV**) and runs Left and Right haptic motors according to closeness to device and position of the obstacle.

We used **dlib computer vision library** to record and extract features to perform **facial recognition**.

**Microsoft Azure Cloud services** takes a series of images to describe the activity taking place.

We used **Optical Character Recognition (Google Cloud)** for Text To Speech Output.

We used **Google Cloud Vision** which classifies and locates the object.

**Twilio API** sends the alert using GPS from Phone when a hazard is detected by the **Google Cloud Vision API**.

QR Scanner scans the QR Code and uses **Capital One API** to make secure and fast transactions in a **BlockChain Network**.

Pulse Sensor data is taken and sent to the server where it is analysed using ML models from **AWS SageMaker** to make the health predictions.

## Challenges we ran into

Making individual modules was a bit easier but integrating them all together into one hardware (Raspberry Pi) and getting them to work was something really challenging to us.

## Accomplishments that we're proud of

The number of features we successfully integrated to prototype level.

## What we learned

We learned to trust in ourselves and our teammates and that when we do that there's nothing we can't accomplish.

## What's next for The Third Eye

Adding a personal assistant to up the game and so much more.
Every person has potential they deserve to unleash; we pledge to level the playfield by taking this initiative forward and strongly urge you to help us in this undertaking.