## 📞 The problem at hand.

How might we leverage an alert agent to ensure **24/7 student safety** from school shootings?

## 🗝️ Where did this stem from?

**School shootings. There are 2 per week**…and this number is NOT decreasing. Students are now worried about their safety but have no way to indicate potential danger. Just two days ago, a suspected shootout at UPenn left witnesses terrified, as suspects made sounds resembling gunshots while exiting. This ongoing sense of fear and uncertainty is unacceptable, especially when lives are at risk.

## 💻 What does Watchful.AI do?

1. **Instant alert** of weapons and suspicious activity, with the versatility and intelligence of GPT-4o
2. Search for and track suspicious activity through video semantic search
3. Monitor multiple video feeds and visualize position of threats in real time
4. **Alert local security** to resolve suspicious activity

## How does it work?

GPT-4o is used in harmony with its distant cousin, CLIP, to progressively distill gigabytes of simultaneous video footage into key anomalies and threat events, which can be efficiently actioned on by a human in the loop. Utilizing CLIP also has the added benefit of enabling semantic search over CCTV footage, so officers can spend less time scrolling through reels of video footage.

## 🛠️ How we built it

**UI/UX Design**
We used Figma and FigJam to lay out the user flow, using research from notable newspaper stations like The Daily Pennsylvanian and CNN to support our visual designs and design system with data.

**Frontend**
Next.js, ShadCN, Tailwind CSS, MappedIn

**Backend**
CLIP (PyTorch + Mac M2), GPT-4o, ChromaDB, FastAPI, OpenCV

## 🕵🏼 Challenges we ran into

A challenge we ran into was collecting data and mapping out the Penn Engineering Building manually. Many tools we wanted to use were unavailable, but we ended up contacting Mappedin and learning their software to create this crucial component of our product

Another significant challenge was utilizing multithreading to processes multiple video streams at a time on a single device. The backend has to ingest, embed, analyze, and serve huge amounts of data.

## ✔️ Accomplishments that we're proud of

We are proud to have built an impactful and well flushed out product in 36 hours! We were able to combine the knowledge of both our frontend team and backend team to work to our strengths and produce a fully functional tool that can be used to prevent an urgent and widespread issue – school shootings.

We were also proud to implement a new technique for cheap but accurate inference on large amounts of video data. Obviously calling GPT-4 for every frame of multiple video streams would be highly cumbersome, but by prescreening with embeddings (which run for free on local GPUs), then checking with GPT-4o, we get the accessibility of on-device AI with the power of huge hosted models.

## ☕️ What we learned

We learned that many students are anxious about school shootings in the US considering there’s an average of 2 shootings per week. Surprisingly, there aren’t any publicly available tools to alert citizens about potential shootings accurately, which is quite concerning. We also learned that we are able to create that tool we wished existed to ensure our safety. Learning how to use MappedIn was something new were discovered to do so.

## ⏭ What's next for Watchful.AI

We strive to implement Watchful.AI in every security system in the US to watch for potential shootouts and suspicious activity. We want to further develop the accuracy of our cameras and description search to ensure safety of not just students, but all citizens in the US.
## Inspiration

While tackling our university courses, we were faced with the task of implementing (and debugging) various data structures. We found that using traditional debuggers or print statements obscures the structure of the data, making testing and debugging code tedious and unintuitive. We were determined to find a way to take advantage of the inherent properties of data structures. With Visually Study Yo Code, you can see the data structure come together as you step through your code.

## What it does

Visually Study Yo Code offers gives you the choice of using a graphical depiction of your variables to debug your JavaScript code. By right clicking a variable in the editor, you can open a tab which displays nodes to depict trees, linked lists and custom data structures. Nodes are added, deleted or modified as you step through your code in the debugger, making it easy to find and trace down errors in your algorithms.

## How we built it

We created and deployed a Visual Studio Code Extension using the Extension API to let us integrate our new functionality into the editor. The graphical representation is constructed using canvas in HTML and displayed to the user by using a webview. To access the data, we interfaced with Visual Studio Code's Debug Adapter and parsed the information about the variables into a JSON object.

## Challenges we ran into

The functionality of our extension was limited by what the Visual Studio Code Extension API provides. While we originally planned to add a command directly to the debugging menu, the ability to add new commands was constrained to the editor. Furthermore, accessing the editors colour themes directly was difficult, so that we decided to limit ourselves to supporting a dark theme.

## Accomplishments that we're proud of

We are proud to have completed a project that we would like to use in the future. In contrast to most hackathon projects, we feel like Visually Study Yo Code benefits us directly by letting us create more robust code more quickly.

## What we learned

Since we had not made an extension before, we learned how to use the Visual Studio Code Extension API. Since Visually Study Yo Code is focused on debugging, we learned about the debugging architecture used by Visual Studio Code.

## What's next for Visually Study Yo Code

Visually Study Yo Code currently only supports debugging in JavaScript. In the future, we will increase the scope of our extension to include other programming languages. Furthermore, we plan to provide support for custom themes for our graphs. By using SVG, we can make our webviews interactive to make debugging even more effective.

## Challenges we ran into

The functionality of our extension was limited by what is possible with the Visual Studio Code Extension API. The ability to add new commands was constrained to the editor, so that we had to give up our original plan of right clicking the variables in the debugging menu. Furthermore, accessing the editors colour themes directly was difficult, so that we decided to limit ourselves to supporting a dark theme for the hackathon.

## Accomplishments that we're proud of

We are proud to have completed a project that we ourselves would like to use in the future. In contrast to most hackathon projects, we feel like Visually Study Yo Code could benefit us in the future by letting us create more robust code more quickly.

## What we learned

We learned how to use the Visual Studio Code Extension API, since we had not made an extension before. Since Visually Study Yo Code is focused on debugging, we learned about the debugging architecture used by Visual Studio Code.

## What's next for Visually Study Yo Code

Visually Study Yo Code currently only supports debugging in JavaScript. In the future, we will increase the scope of our extension to include other programming languages. Furthermore, we plan to provide support for custom themes for our graphs. By using SVG, we can make our webviews interactive to make debugging even more effective.
## Problem Statement

In today’s fast-paced urban environment, family, friends, and emergency responders face significant challenges due to a lack of real-time visual and auditory information. Traditional emergency and SOS systems, such as Life360 and standard 911 calls, primarily rely on voice communication, text descriptions, or geolocation data. However, these methods often lack the situational context needed to fully understand the severity of an incident, placing limitations on what individuals in distress can effectively communicate. Check out our case study below for our stats and modelling.

## Introduction

Guardian is a wearable AI that provides a comprehensive solution integrating real-time video and audio streaming with emotion and object detection analytics. If activated, this data is sent to a dashboard while simultaneously alerting emergency responders and emergency contacts with information about your surroundings and possible threats. By offering a unified platform for live updates and alerts, our project enhances and enables situational awareness—your personal guardian and SOS button.

## Purpose

Guardian is designed with several key objectives in mind to address the limitations of traditional emergency and SOS systems:

* Wearable: Guardian is a discreet and portable device that can be worn comfortably, ensuring that it is always within reach when needed. This increases the likelihood of timely activation during emergencies.
* Hands-free: The system is designed to operate without requiring the user to use their hands, allowing them to focus on the situation at hand. This is particularly important in high-stress or dangerous scenarios where manual operation may not be feasible.
* Capturing Live Data: Guardian continuously captures real-time video and audio data, providing up-to-the-minute information on the user's surroundings. This ensures that emergency responders and contacts receive the most current and relevant situational information.
* Data-Driven Insights: The data collected by Guardian can be used to identify patterns and hotspots, contributing to broader public safety efforts and potentially preventing future incidents.
* Seamless Integration with Emergency Services: Guardian's ability to contact emergency services directly ensures that help is on the way as soon as the device is activated. This integration streamlines the process of seeking assistance in critical moments.

## How we built it

### Hardware

With a Raspberry Pi, we capture video and audio footage using a webcam triggered by a keypad. The keypad serves two functions: enabling Guardian's webcam and contacting 911. Using scripts, the Raspberry Pi then converts the real-time footage into frames with OpenCV to meet machine learning model requirements.

### Backend

We built a deep audio classification CNN model using TensorFlow to classify audio clips as either violent or non-violent based on their spectrograms. This model was trained on over 1000 entries. We also fine-tuned a Hume Expression Measurement model to isolate scores for "aggression," "hostility," and "frustration," outputting a JSON file with the results.

For visual context, we use Gemini flash to identify critical elements from the frames captured by the Raspberry Pi. Authentication and security are managed with Auth0 to ensure secure access to the system.

All collected data is stored in a MongoDB database, allowing for efficient and scalable data management.

### Frontend

We used React, Mapbox, and Three.js to create visual representations of Guardian alerts and crime hotspots. The frontend provides a user-friendly interface for monitoring real-time alerts and viewing historical data on crime hotspots. Users can keep tabs on their loved ones with a 3D informational hub, offering up to date information on emergencies and whereabouts.

## Challenges we ran into

### Implementing the Hardware and Integrating the Raspberry Pi

* Compatibility: Ensuring all components worked seamlessly with the Raspberry Pi.
* Real-time Processing: Optimizing video and audio capture despite limited processing power.
* Power Management: Balancing performance and power consumption for extended operation.

### Integrating Backend, Frontend, and Hardware Components

* Communication: Establishing reliable communication between all components.
* Data Synchronization: Ensuring accurate, real-time data synchronization.
* Security: Implementing secure authentication and encryption with Auth0.

### Fine-tuning and Labeling Datasets for Violent Sounds on Hume AI

* Dataset Labeling: Accurately labeling violent and non-violent audio clips.
* Model Training: Achieving high accuracy in sound classification.
* Emotion Detection: Fine-tuning the model to detect specific emotions.

## Accomplishments that we're proud of

### Hardware

Successfully triggering different functionalities through the keypad, including video capture and emergency calls.

### Frontend

Creating an intuitive and interactive user interface for monitoring alerts and visualizing data.

### Backend

Developing and integrating robust machine learning models for audio and visual classification, and ensuring secure data handling with Auth0 and MongoDB.

## What we learned

We learned how to integrate multiple technologies, including hardware, machine learning models, and secure data management systems, to create a cohesive and effective solution. Here are some key implementations:

### MongoDB

* Scalable Data Management: We learned how to leverage MongoDB’s flexibility and scalability to manage large volumes of real-time data. This included setting up efficient data schemas for storing video frames, audio clips, and emotion analysis results.

### Auth0

* Secure Authentication: Implementing robust authentication mechanisms using Auth0 to protect user data and ensure that only authorized individuals can access sensitive information.
* User Management: Utilizing Auth0’s user management capabilities to handle user roles and permissions, ensuring that different users (e.g., family members, emergency responders) have appropriate access levels.

## What's next for Guardian

Using Guardian we can :

* identify hot spots based off of real-time alert ping locations
* understand emotional cues, assisting emergency responders and guardians in approaching situations
* create a interactive mapping ability that further showcases "safety spots" such as fire stations, police stations, and hospitals
* integration into purses

## Case Study

Conducted a case study of crime hotspots, crime types, and crime per neighbourhoods int Toronto data to fully gauge the problem we are trying to target. Using simple plotting and linear regression models, we took data scraped from the web that identified key statistics:

* 30% of 911 calls could benefit from enhanced situational awareness
* NCMEC reports that inclusion of video footage in Amber Alerts can improve recovery rates by 35%
* The Bureau of Justice Statistics reduced average search time by 40%
* Showing video footage through social media and public alerts increase the number of tips and leads by 50%

Modelling: <https://colab.research.google.com/drive/1M5Ju4ssnh95Qx5bcAweBThjRuZJ34oK4#scrollTo=-_HqXzPZi1eC>
Scraping tool: <https://colab.research.google.com/drive/1iqFYmSIcjy4WgLwTcaR20H9Fcx-SL0lM#scrollTo=Fq4EGEsXqJf1>

## Deployment:

<https://guardian-safety.vercel.app/>

Note: we have not deployed the backends yet! The backends are still local hosted. This is why a lot of the functions aren't working at the moment. Everything (all features) can be found local hosted.

## Github:

<https://github.com/guardiansafety/frontend>

<https://github.com/guardiansafety/backend>

<https://github.com/guardiansafety/hardware>