## Inspiration

Suppose we go out for a run early in the morning without our wallet and cellphone, our service enables banking systems to use facial recognition as a means of payment enabling us to go cashless and cardless.

## What it does

It uses deep neural networks in the back end to detect faces at point of sale terminals and match them with those stored in the banking systems database and lets the customer purchase a product from a verified seller almost instantaneously. In addition, it allows a bill to be divided between customers using recognition of multiple faces. It works in a very non-invasive manner and hence makes life easier for everyone.

## How we built it

Used dlib as the deep learning framework for face detection and recognition, along with Flask for the web API and plain JS on the front end. The front end uses AJAX to communicate with the back end server. All requests are encrypted using SSL (self-signed for the hackathon).

## Challenges we ran into

We attempted to incorporate gesture recognition into the service, but it would cause delays in the transaction due to extensive training/inference based on hand features. This is a feature to be developed in the future, and has the potential to distinguish and popularize our unique service

## Accomplishments that we're proud of

Within 24 hours, we are able to pull up a demo for payment using facial recognition simply by having the customer stand in front of the camera using real-time image streaming. We were also able to enable payment splitting by detection of multiple faces.

## What we learned

We learned to set realistic goals and pivot in the right times. There were points where we thought we wouldn't be able to build anything but we persevered through it to build a minimum viable product. Our lesson of the day would therefore be to never give up and always keep trying -- that is the only reason we could get our demo working by the end of the 24 hour period.

## What's next for GazePay

We plan on associating this service with bank accounts from institutions such as Scotiabank. This will allow users to also see their bank balance after payment, and help us expand our project to include facial recognition ATMs, gesture detection, and voice-enabled payment/ATMs for them to be more accessible and secure for Scotiabank's clients.
## Introduction

Sleep is important, and we all know it. There are many existing products on the market that can assist in your sleep with diverse capabilities and form factors, but they all come at a compromise - smartwatch sleep tracking is bulky and uncomfortable, smart rings serve only as passive trackers, and smart mattresses are exorbitantly expensive.

This is where Silent Night comes in. Our pillow dynamically adjusts to you throughout the night and uses non-intrusive tracking technologies to silently and accurately assess the quality of your sleep.

## What does it do?

The Silent Night pillow comes with three force sensors to track your sleep posture. Based on this information, it adjusts the height of different sections of the pillow with servo motors to enhance your comfort. The pillow also comes equipped with temperature, humidity, and light sensors so that it's aware of any environmental disruptions throughout the night.

Airway collapse (and therein snoring) often occurs when lying down face-up, so we built the servo interaction to suggest the user to sleep on their sides more often. With a built-in microphone, it detects when you are snoring and will change the shape of the pillow to suggest a better-supported sleeping position.

Finally, in the morning, Silent Night provides you with a summary of your sleep quality, giving you insights into your health and full control over your personal data.

## How we built it

The UI was built on Figma and then with HTML, CSS, and Javascript. On the hardware side, we used an ESP32 WROOM 32 module for obtaining sensor data and sending the packaged data as a JSON to a web server. The coding was done in Arduino, taking input from a DHT11 temperature humidity sensor, an electret microphone sensor, three force sensing resistors, and a photoresistor, all wired on a breadboard.

## Challenges we ran into

On the hardware side, we encountered several challenges:

* After connecting to WiFi, we received no data, which was resolved by moving all input pins from ADC2 to ADC1 because all of ADC2 gets used by WiFi.
* The force sensors were not reading workable data until we learned about voltage division and used that resistance information to convert it to workable data.
* Setting up the WiFi network was problematic as we couldn't use eduroam; we ended up using a personal hotspot and configured it to connect to the ESP32.
* We received DMT input as NaN, which was resolved by setting up a catch clause to output previous results if caught.

We also navigated the learning curve of how to code all those sensors in Arduino, send data from the ESP32 to a webserver through WiFi, and package the data in JSON.

On the UI side, converting a complex UI dashboard and integrating it with chart.js was quite difficult. Although we tried to make it work for our live demo, we had to use a backup version of our dashboard instead.

## Accomplishments that we're proud of

We managed to create a fully functional prototype of our idea in 24 hrs, including a hardware and software component. By committing to our idea early on and playing to our strengths, we were able to contribute the most where it mattered, resulting in a project that we are all really proud of.

## What we learned

We gained hands-on experience in integrating various sensors with an ESP32 module, coding in Arduino, and overcoming the challenges of wireless communication and data handling between hardware components and a web server.

## What's next for Silent Night

We'd like to implement an AI-based system that activates the panels only when the user's snoring sound is detected. Along with this, we'd like to reduce the amount of electronics in the pillow and swap it out with inflatable air packets to increase comfort and remove worries of electromagnetic radiation near the head.

### Sources

[https://www.nhlbi.nih.gov/health/sleep-deprivation#:~:text=According%20to%20the%20Centers%20for,at%20least%20once%20a%20month](https://www.nhlbi.nih.gov/health/sleep-deprivation#:%7E:text=According%20to%20the%20Centers%20for,at%20least%20once%20a%20month).
## Inspiration 🍪

We’re fed up with our roommates stealing food from our designated kitchen cupboards. Few things are as soul-crushing as coming home after a long day and finding that someone has eaten the last Oreo cookie you had been saving. Suffice it to say, the university student population is in desperate need of an inexpensive, lightweight security solution to keep intruders out of our snacks...

Introducing **Craven**, an innovative end-to-end pipeline to put your roommates in check and keep your snacks in stock.

## What it does 📸

Craven is centered around a small Nest security camera placed at the back of your snack cupboard. Whenever the cupboard is opened by someone, the camera snaps a photo of them and sends it to our server, where a facial recognition algorithm determines if the cupboard has been opened by its rightful owner or by an intruder. In the latter case, the owner will instantly receive an SMS informing them of the situation, and then our 'security guard' LLM will decide on the appropriate punishment for the perpetrator, based on their snack-theft history. First-time burglars may receive a simple SMS warning, but repeat offenders will have a photo of their heist, embellished with an AI-generated caption, posted on [our X account](https://x.com/craven_htn) for all to see.

## How we built it 🛠️

* **Backend:** Node.js
* **Facial Recognition:** OpenCV, TensorFlow, DLib
* **Pipeline:** Twilio, X, Cohere

## Challenges we ran into 🚩

In order to have unfettered access to the Nest camera's feed, we had to find a way to bypass Google's security protocol. We achieved this by running an HTTP proxy to imitate the credentials of an iOS device, allowing us to fetch snapshots from the camera at any time.

Fine-tuning our facial recognition model also turned out to be a bit of a challenge. In order to ensure accuracy, it was important that we had a comprehensive set of training images for each roommate, and that the model was tested thoroughly. After many iterations, we settled on a K-nearest neighbours algorithm for classifying faces, which performed well both during the day and with night vision.

Additionally, integrating the X API to automate the public shaming process required specific prompt engineering to create captions that were both humorous and effective in discouraging repeat offenders.

## Accomplishments that we're proud of 💪

* Successfully bypassing Nest’s security measures to access the camera feed.
* Achieving high accuracy in facial recognition using a well-tuned K-nearest neighbours algorithm.
* Fine-tuning Cohere to generate funny and engaging social media captions.
* Creating a seamless, rapid security pipeline that requires no legwork from the cupboard owner.

## What we learned 🧠

Over the course of this hackathon, we gained valuable insights into how to circumvent API protocols to access hardware data streams (for a good cause, of course). We also deepened our understanding of facial recognition technology and learned how to tune computer vision models for improved accuracy. For our X integration, we learned how to engineer prompts for Cohere's API to ensure that the AI-generated captions were both humorous and contextual. Finally, we gained experience integrating multiple APIs (Nest, Twilio, X) into a cohesive, real-time application.

## What's next for Craven 🔮

* **Multi-owner support:** Extend Craven to work with multiple cupboards or fridges in shared spaces, creating a mutual accountability structure between roommates.
* **Machine learning improvement:** Experiment with more advanced facial recognition models like deep learning for even better accuracy.
* **Social features:** Create an online leaderboard for the most frequent offenders, and allow users to vote on the best captions generated for snack thieves.
* **Voice activation:** Add voice commands to interact with Craven, allowing roommates to issue verbal warnings when the cupboard is opened.