## Inspiration

One In every 250 people suffer from cerebral palsy, where the affected person cannot move a limb properly, And thus require constant care throughout their lifetimes. To ease their way of living, we have made this project, 'para-pal'.

The inspiration for this idea was blended with a number of research papers and a project called Pupil which used permutations to make communication possible with eye movements.

## What it does

![Main](https://media.discordapp.net/attachments/828211308305448983/828261879326572544/iris_seg.png?width=819&height=355)

**"What if Eyes can Speak? Yesss - you heard it right!"**
Para-pal is a novel idea that tracks patterns in the eye movement of the patient and then converts into actual speech. We use the state-of-the-art iris recognition (dlib) to accurately track the eye movements to figure out the the pattern. Our solution is sustainable and very cheap to build and setup. Uses QR codes to connect the caretaker and the patient's app.

We enable paralyzed patients to **navigate across the screen using their eye movements**. They can select an action by placing the cursor for more than 3 seconds or alternatively, they can **blink three times to select the particular action**. A help request is immediately sent to the mobile application of the care taker as a **push notification**

## How we built it

We've embraced flutter in our frontend to make the UI - simple, intuitive with modularity and customisabilty. The image processing and live-feed detection are done on a separate child python process. The iris-recognition at it's core uses dlib and pipe the output to opencv.
We've developed a desktop-app (which is cross-platform with a rpi3 as well)for the patient and a mobile app for the caretaker.

We also tried running our desktop application on Raspberry Pi using an old laptop screen. In the future, we wish to make a dedicated hardware which can be cost-efficient for patients with paralysis.

![hardware](https://media.discordapp.net/attachments/828211308305448983/828263070228676638/20210404_191100.jpg?width=542&height=406)

![hardware2](https://media.discordapp.net/attachments/828211308305448983/828263051420762182/20210404_191120.jpg?width=542&height=406)

## Challenges we ran into

Building up the dlib took a significant amount of time, because there were no binaries/wheels and we had to build from source. Integrating features to enable connectivity and sessions between the caretaker's mobile and the desktop app was hard. Fine tuning some parameters of the ML model, preprocessing and cleaning the input was a real challenge.

Since we were from a different time zone, it was challenging to stay awake throughout the 36 hours and make this project!

## Accomplishments that we're proud of

* An actual working application in such a short time span.
* Integrating additional hardware of a tablet for better camera accuracy.
* Decoding the input feed with a very good accuracy.
* Making a successful submission for HackPrinceton.
* Team work :)

## What we learned

* It is always better to use a pre-trained model than making one yourself, because of the significant accuracy difference.
* QR scanning is complex and is harder to integrate in flutter than how it looks on the outside.
* Rather than over-engineering a flutter component, search if a library exists that does exactly what is needed.

## What's next for Para Pal - What if your eyes can speak?

* More easier prefix-less code patterns for the patient using an algorithm like huffman coding.
* More advanced controls using ML that tracks and learns the patient's regular inputs to the app.
* Better analytics to the care-taker.
* More UI colored themes.
## Inspiration

We wanted to make the interactions with our computers more intuitive while giving people with special needs more options to navigate in the digital world. With the digital landscape around us evolving, we got inspired by scenes in movies featuring Tony Stark, where he interacts with computers within his high-tech office. Instead of using a mouse and computer, he uses hand gestures and his voice to control his work environment.

## What it does

Instead of a mouse, Input/Output Artificial Intelligence, or I/OAI, uses a user's webcam to move their cursor to where their face OR hand is pointing towards through machine learning.

Additionally, I/OAI allows users to map their preferred hand movements for commands such as "click", "minimize", "open applications", "navigate websites", and more!

I/OAI also allows users to input data using their voice, so they don't need to use a keyboard and mouse. This increases accessbility for those who don't readily have access to these peripherals.

## How we built it

Face tracker -> Dlib

Hand tracker -> Mediapipe

Voice Recognition -> Google Cloud

Graphical User Interface -> tkinter

Mouse and Keyboard Simulation -> pyautogui

## Challenges we ran into

Running this many programs at the same time slows it down considerably, we therefore need to selectively choose which ones we wanted to keep during the implementation. We solved this by using multithreading and carefully investigating efficiency.

We also had a hard time mapping the face because of the angles of rotation of the head, increasing the complexity of the matching algorithm.

## Accomplishments we're proud of

We were able to implement everything we set out to do in a short amount of time, as there was a lot of integrations with multiple frameworks and our own algorithms.

## What we learned

How to use multithreading for multiple trackers, using openCV for easy camera frames, tkinter GUI building and pyautogui for automation.

## What's next for I/OAI

We need to figure a way to incorporate features more efficiently or get a supercomputer like Tony Stark!

By improving on the features, people will have more accessbility at their computers by simply downloading a program instead of buying expensive products like an eyetracker.
## Inspiration

Alzheimer's is a very interesting, scary disease. There's no decided cause for the neuro-degenerative disease among scientists, and neither is there any cure. And on a personal level, we've all met people before who have had family members either currently or previously suffering from Alzheimer's. Inspired by this and the health challenges posted at this hackathon, we decided to come up with a tool to help with Alzheimer's diagnosis and recovery efforts.

## What it does

MindTune is a tool to detect early signs of Alzheimer's and other neuro-degenerative diseases in patients. At-risk patients download the app on their phone and participate in an in-app session on a weekly or daily basis. This session has 2 main features: a comprehensive Neuro-Cognition testing system and an accurate Eye-Tracking system. Doctors can then view this data in a separate website and analyze their patients for any early signs of diseases. Our Neuro-Cognition system works by prompting users questions about their life/day and then using NLP to analyze their responses for any warning signs of reduced cognitive ability. Our Eye-Tracking system uses a highly-accurate gaze-estimation model to check "jiteriness" in users' pupils as another warning sign of Alzheimer's.

## How we built it

MindTune was built with a lot of different tools, primarily:

* React Native - frontend development and user interactivity
* OpenCV - pupil-detection and movement tracking
* PyTorch - run inference with model
* Modal Labs - hosting and deploying
* FastAPI/Python - backend server, LLM integration, NLP

## Challenges we ran into

One of the major challenges we ran into was getting eye-tracking to work in real-time. At first, we developed our own pupil-recognition system with OpenCV. However, this system was slow and couldn't be effectively used in the app. We moved over to a new model (gaze-estimation) and invested heavy effort into making it run as fast as possible with low latency. Another challenge was designing the neuro-cognition system. The questions our system asked had to be relevant to users while still staying semi-randomized.

## Accomplishments that we're proud of

We're really proud of our eye-tracking system. We researched links between pupil movements and Alzheimer's to find the best method of detecting warning signs through pupil jitteriness. We're also proud of our accessible UI, meant to be easy to use by elderly patients, caretakers, and doctors.

## What we learned

We learned a lot about facial recognition and utilizing OpenCV for such recognition tasks. We also learned about using React Native for front-end app development.

## What's next for MindTune

In the future, we want to add even more cognitive tests and puzzles to help find more warning signs in users. We also want to expand the doctor portal to have more advanced analysis of patient data.