## Inspiration

Over 50 million people suffer from epilepsy worldwide, making it one of the most common neurological diseases. With the rise in digital adoption, people who suffer from epilepsy are at risk from online videos that may trigger seizure responses. This can have adverse health effects on the lives of epileptic individuals and inhibits their interaction with technology. Often, there are warnings on videos that may trigger seizures, however, not much effort is made to resolve the problem. Our goal is clear, to provide a solution that proactively solves the problem and increases accessibility for this target group.

## What it does

We built a chrome extension that interacts with YouTube videos and provides users with a warning in advance of seizure-inducing content and applies a filter to allow people to continue watching the video.

## How we built it

We first started off by building our flask server that has an open endpoint where the youtube video url is passed. This video is then downloaded to our server using PyTube and passed to openCv which determines the luminance values across the video. This dataset is then pre-processed and passed to the Azure Anomaly Detection model which determines anomalies in the luminance dataset. These anomalies represent portions in the youtube video where there are large discrepancies in light variation and hence, could trigger potential seizures for photosensitive epileptic users. The anomaly dataset is then processed to determine the timestamps in the video where these events occur. This dataset is then returned to a google chrome extension which overlays a dark filter over the video during these anomaly timestamps, thus preventing a potential seizure.

## Challenges we ran into

The first major challenge we faced was attempting to use Google Cloud Services as our anomaly detector for our luminance dataset as this service required us to package the data into a model and then create an anomaly detection model. As we were inexperienced in this subject, we were unable to complete this objective, this prevented us from using Google Cloud Services and caused us to lose precious development time. Another challenge that we faced was with the framerate of our downloaded videos. In our first iteration of creating the mvp, we were originally processing the youtube videos at a frame rate of 8 frames per second while creating the video timestamps on the anomaly data at a frame rate of 30 frames per second. This caused the extension to overlay the seizure warning filter at incorrect times of the video. Once we noticed that we were incorrectly downloading the videos at a slower frame rate, we were able to swiftly fix this issue.

## What's next for Epilepsy Safe Viewer

The first thing that we wanted to do as a team was some form of user testing on the product to determine the effectiveness of the product while also determining potential improvement areas. This would allow us to reiterate through our design process and make the product even better at solving our targeted user group’s problem. We also wanted to try applying our extension to users that face startle epilepsy by determining anomalies in the audio decibel levels of Youtube videos and normalizing the audio based on this. Lastly, we wanted to look into potential opportunities of expanding this product to other video platforms such as Netflix and TikTok, thus increasing the inclusivity and accessibility of this user group with technology.
## Inspiration

The inspiration for *SurgeVue* came from the high risks associated with brain surgeries, where precision is critical and the margin for error is minimal. With a mortality rate that increases 15-fold for non-elective neurosurgeries, we were motivated to create a solution that leverages augmented reality (AR) and machine learning to enhance the precision of brain tumor removal surgeries. By merging real-time data with surgical tools, we aimed to provide surgeons with an advanced assistant to reduce risks and save lives.

## What it does

*SurgeVue* is an AR-powered surgical assistant that provides neurosurgeons with real-time visual overlays during brain surgery. Using machine learning, the system outlines tumors on the patient's brain and classifies the presence of foreign objects. Integrated with an Arduino gyroscope for hand tracking, SurgeVue offers surgeons real-time feedback on tool movements, sweat sensor data, and critical hand movement trends—all displayed within a secure mobile app that ensures patient data privacy using facial recognition and RFID technology. The system empowers surgeons to make more informed, precise decisions during the most delicate procedures.

## How we built it

We built *SurgeVue* using a combination of cutting-edge technologies:

* **OpenCV** for real-time tumor detection and hand movement detection for augmented view.
* **PyTorch** for classifying tumors.
* **Swift and SceneKit** to create an immersive AR environment that overlays tumor outlines onto the surgeon's view.
* **Arduino gyroscope** for tracking the surgeon's hand movements and tool positioning.
* **PropelAuth** to ensure secure access to sensitive patient data via facial recognition and RFID.
* **Flask backend** to process machine learning models and serve image classification results via API.
* **Mobile App** that visualizes gyroscope, sweat sensor, and hand movement trends.

## Challenges we ran into

One of the biggest challenges was ensuring that the AR overlay, tumor detection, and hand-tracking happened in real-time without latency. We had to optimize our models to ensure seamless performance in the fast-paced environment of an operating room. Integrating the hardware components like the Arduino gyroscope and managing precise hand-tracking also posed challenges, as did creating a user-friendly interface that was informative without being overwhelming during a surgery.

## Accomplishments that we're proud of

* Successfully implementing real-time AR overlays that provide surgeons with critical information at a glance.
* Developing a machine learning model that accurately classifies tumors and detects foreign objects.
* Integrating hardware sensors (gyroscope, sweat sensors) to provide surgeons with hand movement insights, enhancing precision during surgeries.
* Ensuring patient data security through advanced authentication measures like facial recognition and RFID.

## What we learned

We learned how to combine AR and machine learning into a cohesive solution that can operate in real-time under intense conditions like surgery. We also gained experience in integrating hardware components, optimizing machine learning models for low latency, and handling large datasets like medical imaging. Furthermore we can save OR nurses and surgeons from intense radiation from the Medtronic devices, ones that prevent them from continuing operation. Additionally, building an intuitive, non-intrusive interface for surgeons highlighted the importance of user-centered design in healthcare applications.

## What's next for SurgeVue

Next, we plan to:

* **Refine the Machine Learning Model**: Enhance tumor classification accuracy and expand it to detect other conditions and anomalies.
* **Clinical Trials**: Test *SurgeVue* in real-world surgical settings and gather feedback from neurosurgeons.
* **Tool Tracking**: Further refine the hand-tracking and integrate more advanced surgical tools into the AR environment.
* **Global Expansion**: Implement support for other AR platforms like Hololens and explore expanding the use of the system in other complex surgeries beyond neurosurgery.
* **3D Implementation**: Create 3D models of the brain for the surgeon to interact with in real-time
## Inspiration

There are approximately **10 million** Americans who suffer from visual impairment, and over **5 million Americans** suffer from Alzheimer's dementia. This weekend our team decided to help those who were not as fortunate. We wanted to utilize technology to create a positive impact on their quality of life.

## What it does

We utilized a smartphone camera to analyze the surrounding and warn visually impaired people about obstacles that were in their way. Additionally, we took it a step further and used the **Azure Face API** to detect the faces of people that the user interacted with and we stored their name and facial attributes that can be recalled later. An Alzheimer's patient can utilize the face recognition software to remind the patient of who the person is, and when they last saw him.

## How we built it

We built our app around **Azure's APIs**, we created a **Custom Vision** network that identified different objects and learned from the data that we collected from the hacking space. The UI of the iOS app was created to be simple and useful for the visually impaired, so that they could operate it without having to look at it.

## Challenges we ran into

Through the process of coding and developing our idea we ran into several technical difficulties. Our first challenge was to design a simple UI, so that the visually impaired people could effectively use it without getting confused. The next challenge we ran into was attempting to grab visual feed from the camera, and running them fast enough through the Azure services to get a quick response. Another challenging task that we had to accomplish was to create and train our own neural network with relevant data.

## Accomplishments that we're proud of

We are proud of several accomplishments throughout our app. First, we are especially proud of setting up a clean UI with two gestures, and voice control with speech recognition for the visually impaired. Additionally, we are proud of having set up our own neural network, that was capable of identifying faces and objects.

## What we learned

We learned how to implement **Azure Custom Vision and Azure Face APIs** into **iOS**, and we learned how to use a live camera feed to grab frames and analyze them. Additionally, not all of us had worked with a neural network before, making it interesting for the rest of us to learn about neural networking.

## What's next for BlindSpot

In the future, we want to make the app hands-free for the visually impaired, by developing it for headsets like the Microsoft HoloLens, Google Glass, or any other wearable camera device.