## Problem Statement

In today’s fast-paced urban environment, family, friends, and emergency responders face significant challenges due to a lack of real-time visual and auditory information. Traditional emergency and SOS systems, such as Life360 and standard 911 calls, primarily rely on voice communication, text descriptions, or geolocation data. However, these methods often lack the situational context needed to fully understand the severity of an incident, placing limitations on what individuals in distress can effectively communicate. Check out our case study below for our stats and modelling.

## Introduction

Guardian is a wearable AI that provides a comprehensive solution integrating real-time video and audio streaming with emotion and object detection analytics. If activated, this data is sent to a dashboard while simultaneously alerting emergency responders and emergency contacts with information about your surroundings and possible threats. By offering a unified platform for live updates and alerts, our project enhances and enables situational awareness—your personal guardian and SOS button.

## Purpose

Guardian is designed with several key objectives in mind to address the limitations of traditional emergency and SOS systems:

* Wearable: Guardian is a discreet and portable device that can be worn comfortably, ensuring that it is always within reach when needed. This increases the likelihood of timely activation during emergencies.
* Hands-free: The system is designed to operate without requiring the user to use their hands, allowing them to focus on the situation at hand. This is particularly important in high-stress or dangerous scenarios where manual operation may not be feasible.
* Capturing Live Data: Guardian continuously captures real-time video and audio data, providing up-to-the-minute information on the user's surroundings. This ensures that emergency responders and contacts receive the most current and relevant situational information.
* Data-Driven Insights: The data collected by Guardian can be used to identify patterns and hotspots, contributing to broader public safety efforts and potentially preventing future incidents.
* Seamless Integration with Emergency Services: Guardian's ability to contact emergency services directly ensures that help is on the way as soon as the device is activated. This integration streamlines the process of seeking assistance in critical moments.

## How we built it

### Hardware

With a Raspberry Pi, we capture video and audio footage using a webcam triggered by a keypad. The keypad serves two functions: enabling Guardian's webcam and contacting 911. Using scripts, the Raspberry Pi then converts the real-time footage into frames with OpenCV to meet machine learning model requirements.

### Backend

We built a deep audio classification CNN model using TensorFlow to classify audio clips as either violent or non-violent based on their spectrograms. This model was trained on over 1000 entries. We also fine-tuned a Hume Expression Measurement model to isolate scores for "aggression," "hostility," and "frustration," outputting a JSON file with the results.

For visual context, we use Gemini flash to identify critical elements from the frames captured by the Raspberry Pi. Authentication and security are managed with Auth0 to ensure secure access to the system.

All collected data is stored in a MongoDB database, allowing for efficient and scalable data management.

### Frontend

We used React, Mapbox, and Three.js to create visual representations of Guardian alerts and crime hotspots. The frontend provides a user-friendly interface for monitoring real-time alerts and viewing historical data on crime hotspots. Users can keep tabs on their loved ones with a 3D informational hub, offering up to date information on emergencies and whereabouts.

## Challenges we ran into

### Implementing the Hardware and Integrating the Raspberry Pi

* Compatibility: Ensuring all components worked seamlessly with the Raspberry Pi.
* Real-time Processing: Optimizing video and audio capture despite limited processing power.
* Power Management: Balancing performance and power consumption for extended operation.

### Integrating Backend, Frontend, and Hardware Components

* Communication: Establishing reliable communication between all components.
* Data Synchronization: Ensuring accurate, real-time data synchronization.
* Security: Implementing secure authentication and encryption with Auth0.

### Fine-tuning and Labeling Datasets for Violent Sounds on Hume AI

* Dataset Labeling: Accurately labeling violent and non-violent audio clips.
* Model Training: Achieving high accuracy in sound classification.
* Emotion Detection: Fine-tuning the model to detect specific emotions.

## Accomplishments that we're proud of

### Hardware

Successfully triggering different functionalities through the keypad, including video capture and emergency calls.

### Frontend

Creating an intuitive and interactive user interface for monitoring alerts and visualizing data.

### Backend

Developing and integrating robust machine learning models for audio and visual classification, and ensuring secure data handling with Auth0 and MongoDB.

## What we learned

We learned how to integrate multiple technologies, including hardware, machine learning models, and secure data management systems, to create a cohesive and effective solution. Here are some key implementations:

### MongoDB

* Scalable Data Management: We learned how to leverage MongoDB’s flexibility and scalability to manage large volumes of real-time data. This included setting up efficient data schemas for storing video frames, audio clips, and emotion analysis results.

### Auth0

* Secure Authentication: Implementing robust authentication mechanisms using Auth0 to protect user data and ensure that only authorized individuals can access sensitive information.
* User Management: Utilizing Auth0’s user management capabilities to handle user roles and permissions, ensuring that different users (e.g., family members, emergency responders) have appropriate access levels.

## What's next for Guardian

Using Guardian we can :

* identify hot spots based off of real-time alert ping locations
* understand emotional cues, assisting emergency responders and guardians in approaching situations
* create a interactive mapping ability that further showcases "safety spots" such as fire stations, police stations, and hospitals
* integration into purses

## Case Study

Conducted a case study of crime hotspots, crime types, and crime per neighbourhoods int Toronto data to fully gauge the problem we are trying to target. Using simple plotting and linear regression models, we took data scraped from the web that identified key statistics:

* 30% of 911 calls could benefit from enhanced situational awareness
* NCMEC reports that inclusion of video footage in Amber Alerts can improve recovery rates by 35%
* The Bureau of Justice Statistics reduced average search time by 40%
* Showing video footage through social media and public alerts increase the number of tips and leads by 50%

Modelling: <https://colab.research.google.com/drive/1M5Ju4ssnh95Qx5bcAweBThjRuZJ34oK4#scrollTo=-_HqXzPZi1eC>
Scraping tool: <https://colab.research.google.com/drive/1iqFYmSIcjy4WgLwTcaR20H9Fcx-SL0lM#scrollTo=Fq4EGEsXqJf1>

## Deployment:

<https://guardian-safety.vercel.app/>

Note: we have not deployed the backends yet! The backends are still local hosted. This is why a lot of the functions aren't working at the moment. Everything (all features) can be found local hosted.

## Github:

<https://github.com/guardiansafety/frontend>

<https://github.com/guardiansafety/backend>

<https://github.com/guardiansafety/hardware>
## 💡Inspiration and Purpose

Our group members are from all around the U.S., from Dallas to Boston, but one thing that we have in common is that we have a grandparent living by themselves. Due to various family situations, we have elderly loved ones who spend a few hours alone each day, putting them at risk of a fatal accident like a fall. After researching current commercial solutions for fall detection, devices can range from anywhere from twenty-four to fifty dollars per device monthly. While this is affordable for our families, it is distinctly out of reach for many disadvantaged families.

In 2020, falls among Americans 65 and older caused over 36,000 deaths - the leading cause of injury death for that demographic group. Furthermore, these falls led to some three million emergency hospital visits and $50 billion in medical costs, creating an immense strain on our healthcare system.

## 🎖️ What It Does

Despite these sobering statistics, very few accessible solutions exist to assist seniors who may experience a fall at home. More worryingly, many of these products rely on the individual who has just become seriously injured to report the incident and direct emergency medical services to their location. Oftentimes, someone might not be able to alert the proper authorities and suffer needlessly as a result.

Guardian Angel aims to solve this growing problem by automating the process of fall detection using machine learning and deploying it in an easy-to-use, accurate, and portable system. From there, the necessary authorities can be notified, even if an individual is critically injured.

By using data-based trackers such as position, this application will adequately calculate and determine if a person has fallen based on a point and confidence system. This can be used for risk prevention profiles as it can detect shuffling or skipping. The application will shine red if it is confident that the person has fallen and it is not a false positive.

## 🧠 Development and Challenges

The application overall was built using React, alongside TailwindCSS and ML5. Using the create-react-app boilerplate, we were able to minimize the initial setup and invest more time in fixing bugs. In general, the website works by obtaining the model, metadata, and weights from the tensorflow.js model, requests a webcam input, and then pushes that webcam input into the machine learning model. The result of the model, as well as the confidence, is then displayed to the user.

The machine learning side of the project was developed using TensorFlow through tensorflow.js. As a result, we were able to primarily focus on gathering proper samples for the model, rather than optimization. The samples used for training were gathered by capturing dozens of videos of people falling in various backgrounds, isolating the frames, and manually labeling the data. We also augmented the samples to improve the model's ability to generalize.

For training, we used around 5 epochs, a learning rate of 0.01, and a batch size of 16. Other than that, every other parameter was default.

We also ran into a few major challenges during the creation of the project. Firstly, the complexity of detecting movement made obtaining samples and training particularly difficult. Our solution was to gather a number of samples from different orientations, allowing for the model to be accurate in a wide variety of situations. Specifically speaking for the website, one of the main challenges we ran implementing video rendering within React. We were eventually able to implement our desired functionality using HTML videos, uncommon react dependency components, and complex Tensorflow-React interactions.

## 🎖️ Accomplishments that we're proud of

First and foremost, we are proud of successfully completing the project while getting to meet with several mentors and knowledgeable professionals, such as the great teams at CareYaya and Y Combinator.

Second, we are proud of our model's accuracy. Classifying a series of images with movement was no small undertaking, particularly because we needed to ignore unintentional movements while continuing to accurately identify sudden changes in motion.

Lastly, we are very proud of the usability and functionality of our application. Despite being a very complex undertaking, our front end wraps up our product into an incredibly easy-to-use platform that is highly intuitive. We hope that this project, with minimal modifications, could be deployed commercially and begin to make a difference.

## 🔜 Next Steps

We would soon like to integrate our app with smart devices such as Amazon Alexa and Apple Smart Watch to obtain more data and verify our predictions. With accessible devices like Alexa already on the market, we will continue to ensure this service is as scalable as possible. We also want to detect fall audio, as audio can help make our results more accurate. Audio also would allow our project to be active in more locations in a household, and cover for the blind spots cameras typically have.

Guardian Angel is a deployable and complete application in itself, but we hope to take extra steps to make our project even more user-friendly. One thing we want to implement in the near future is to create an API for our application so that we can take advantage of existing home security systems. By utilizing security cameras that are already present, we can lower the barriers to entry for consumers as well as improve our reach.
## Problem Statement

As the number of the elderly population is constantly growing, there is an increasing demand for home care. In fact, the market for safety and security solutions in the healthcare sector is estimated to reach $40.1 billion by 2025.
The elderly, disabled, and vulnerable people face a constant risk of falls and other accidents, especially in environments like hospitals, nursing homes, and home care environments, where they require constant supervision. However, traditional monitoring methods, such as human caregivers or surveillance cameras, are often not enough to provide prompt and effective responses in emergency situations. This potentially has serious consequences, including injury, prolonged recovery, and increased healthcare costs.

## Solution

The proposed app aims to address this problem by providing real-time monitoring and alert system, using a camera and cloud-based machine learning algorithms to detect any signs of injury or danger, and immediately notify designated emergency contacts, such as healthcare professionals, with information about the user's condition and collected personal data.
We believe that the app has the potential to revolutionize the way vulnerable individuals are monitored and protected, by providing a safer and more secure environment in designated institutions.

## Developing Process

Prior to development, our designer used Figma to create a prototype which was used as a reference point when the developers were building the platform in HTML, CSS, and ReactJs.
For the cloud-based machine learning algorithms, we used Computer Vision, Open CV, Numpy, and Flask to train the model on a dataset of various poses and movements and to detect any signs of injury or danger in real time.
Because of limited resources, we decided to use our phones as an analogue to cameras to do the live streams for the real-time monitoring.

## Impact

* **Improved safety:** The real-time monitoring and alert system provided by the app helps to reduce the risk of falls and other accidents, keeping vulnerable individuals safer and reducing the likelihood of serious injury.
* **Faster response time:** The app triggers an alert and sends notifications to designated emergency contacts in case of any danger or injury, which allows for a faster response time and more effective response.
* **Increased efficiency:** Using cloud-based machine learning algorithms and computer vision techniques allow the app to analyze the user's movements and detect any signs of danger without constant human supervision.
* **Better patient care:** In a hospital setting, the app could be used to monitor patients and alert nurses if they are in danger of falling or if their vital signs indicate that they need medical attention. This could lead to improved patient care, reduced medical costs, and faster recovery times.
* **Peace of mind for families and caregivers:** The app provides families and caregivers with peace of mind, knowing that their loved ones are being monitored and protected and that they will be immediately notified in case of any danger or emergency.

## Challenges

One of the biggest challenges have been integrating all the different technologies, such as live streaming and machine learning algorithms, and making sure they worked together seamlessly.

## Successes

The project was a collaborative effort between a designer and developers, which highlights the importance of cross-functional teams in delivering complex technical solutions. Overall, the project was a success and resulted in a cutting-edge solution that can help protect vulnerable individuals.

## Things Learnt

* **Importance of cross-functional teams:** As there were different specialists working on the project, it helped us understand the value of cross-functional teams in addressing complex challenges and delivering successful results.
* **Integrating different technologies:** Our team learned the challenges and importance of integrating different technologies to deliver a seamless and effective solution.
* **Machine learning for health applications:** After doing the research and completing the project, our team learned about the potential and challenges of using machine learning in the healthcare industry, and the steps required to build and deploy a successful machine learning model.

## Future Plans for SafeSpot

* First of all, the usage of the app could be extended to other settings, such as elderly care facilities, schools, kindergartens, or emergency rooms to provide a safer and more secure environment for vulnerable individuals.
* Apart from the web, the platform could also be implemented as a mobile app. In this case scenario, the alert would pop up privately on the user’s phone and notify only people who are given access to it.
* The app could also be integrated with wearable devices, such as fitness trackers, which could provide additional data and context to help determine if the user is in danger or has been injured.