## Inspiration

There were two primary sources of inspiration. The first one was a paper published by University of Oxford researchers, who proposed a state of the art deep learning pipeline to extract spoken language from video. The paper can be found [here](http://www.robots.ox.ac.uk/%7Evgg/publications/2018/Afouras18b/afouras18b.pdf). The repo for the model used as a base template can be found [here](https://github.com/afourast/deep_lip_reading).

The second source of inspiration is an existing product on the market, [Focals by North](https://www.bynorth.com/). Focals are smart glasses that aim to put the important parts of your life right in front of you through a projected heads up display. We thought it would be a great idea to build onto a platform like this through adding a camera and using artificial intelligence to gain valuable insights about what you see, which in our case, is deciphering speech from visual input. This has applications in aiding individuals who are deaf or hard-of-hearing, noisy environments where automatic speech recognition is difficult, and in conjunction with speech recognition for ultra-accurate, real-time transcripts.

## What it does

The user presses a button on the side of the glasses, which begins recording, and upon pressing the button again, recording ends. The camera is connected to a raspberry pi, which is a web enabled device. The raspberry pi uploads the recording to google cloud, and submits a post to a web server along with the file name uploaded. The web server downloads the video from google cloud, runs facial detection through a haar cascade classifier, and feeds that into a transformer network which transcribes the video. Upon finished, a front-end web application is notified through socket communication, and this results in the front-end streaming the video from google cloud as well as displaying the transcription output from the back-end server.

## How we built it

The hardware platform is a raspberry pi zero interfaced with a pi camera. A python script is run on the raspberry pi to listen for GPIO, record video, upload to google cloud, and post to the back-end server. The back-end server is implemented using Flask, a web framework in Python. The back-end server runs the processing pipeline, which utilizes TensorFlow and OpenCV. The front-end is implemented using React in JavaScript.

## Challenges we ran into

* TensorFlow proved to be difficult to integrate with the back-end server due to dependency and driver compatibility issues, forcing us to run it on CPU only, which does not yield maximum performance
* It was difficult to establish a network connection on the Raspberry Pi, which we worked around through USB-tethering with a mobile device

## Accomplishments that we're proud of

* Establishing a multi-step pipeline that features hardware, cloud storage, a back-end server, and a front-end web application
* Design of the glasses prototype

## What we learned

* How to setup a back-end web server using Flask
* How to facilitate socket communication between Flask and React
* How to setup a web server through local host tunneling using ngrok
* How to convert a video into a text prediction through 3D spatio-temporal convolutions and transformer networks
* How to interface with Google Cloud for data storage between various components such as hardware, back-end, and front-end

## What's next for Synviz

* With stronger on-board battery, 5G network connection, and a computationally stronger compute server, we believe it will be possible to achieve near real-time transcription from a video feed that can be implemented on an existing platform like North's Focals to deliver a promising business appeal
## Inspiration

We were inspired by how there were many instances of fraud with regards to how donations were handled. It is heartbreaking to see how mismanagement can lead to victims of disasters not receiving the help that they desperately need.

## What it does

TrustTrace introduces unparalleled levels of transparency and accountability to charities and fundraisers. Donors will now feel more comfortable donating to causes such as helping earthquake victims since they will now know how their money will be spent, and where every dollar raised is going to.

## How we built it

We created a smart contract that allowed organisations to specify how much money they want to raise and how they want to segment their spending into specific categories. This is then enforced by the smart contract, which puts donors at ease as they know that their donations will not be misused, and will go to those who are truly in need.

## Challenges we ran into

The integration of smart contracts and the web development frameworks were more difficult than we expected, and we overcame them with lots of trial and error.

## Accomplishments that we're proud of

We are proud of being able to create this in under 36 hours for a noble cause. We hope that this creation can eventually evolve into a product that can raise interest, trust, and participation in donating to humanitarian causes across the world.

## What we learned

We learnt a lot about how smart contract works, how they are deployed, and how they can be used to enforce trust.

## What's next for TrustTrace

Expanding towards serving a wider range of fundraisers such as GoFundMes, Kickstarters, etc.
## Inspiration

The healthcare industry often struggles with challenges related to data security, management, and interoperability between hospitals. **ConnectHealth** was created to address these challenges by:

1. **Ensuring Data Security**: With sensitive information such as personal, medical, and financial data, the platform ensures all data is securely stored and encrypted to protect patient privacy.
2. **Improving Hospital Collaboration**: Many hospitals face difficulties sharing patient information across different facilities. ConnectHealth connects hospitals on a unified platform, allowing seamless and secure sharing of patient data.
3. **Introducing AI-Assisted Decision Making**: The integration of **Ruff**, our AI assistant, helps healthcare professionals by offering real-time suggestions on medications and treatments, speeding up decision-making and improving patient outcomes.
4. **Simplifying Data Management**: ConnectHealth offers an easy-to-use interface that allows healthcare providers to manage medical records, prescriptions, and financial information efficiently, reducing administrative overhead and ensuring compliance with healthcare regulations.

## What it does

# Ruff: Your Friendly Medical Assistant 🐶

Ruff is a helpful and interactive feature designed to assist healthcare professionals with quick access to patient data and medical advice. Built using React, Firebase, and Axios, Ruff operates through a chat interface where users can type commands to retrieve specific information or get advice.

### Key Features:

* **Find Patient Information**: Command `find {user name}` to search for a patient’s details like name, DOB, allergies, address, and phone number.
* **Retrieve Medical Records**: Command `medical {user name}` to pull patient records, including prescription history and medical visits.
* **Access Patient Profile URL**: Use `go to {user name}` for a direct link to the patient's profile.
* **General Commands**: Typing `help` shows available commands and how Ruff can assist with medical advice or data retrieval.

Ruff is a lighthearted, dog-friendly companion, providing quick and accurate information!

# Key Features

### 1. **Personalized Patient Management**

* **Comprehensive Patient Profiles**: Manage personal data like name, DOB, and contacts.
* **Photo and Document Uploads**: Securely store patient photos and insurance documents.
* **Editable Information**: Update patient data with validation.

### 2. **Medical Records and Visits**

* **Visit Tracking**: Track visits, prescriptions, and outcomes.
* **Prescription Management**: Prescribe, track, and update medications.
* **AI Suggestions**: Ruff suggests medications based on history and symptoms.

### 3. **Secure Financial Data**

* **Credit Card Management**: Securely manage patient card information.
* **Billing Integration**: Track bills and integrate insurance providers.

### 4. **Easy Importion**

* **Seamless Data Import**: Easily import patient data from external sources, ensuring consistency and accuracy.
* **Validation and Review**: Imported data is automatically validated and presented for review before finalizing the process.

## How we built it

![infastructor](https://github.com/user-attachments/assets/95fa3e45-7921-4ad1-bb03-a66bf1a70b5f)

## Technology Stack

* **Frontend**: React.js
* **Backend**: Node.js
* **Database**: Firebase Firestore
* **AI Integration**: Ruff AI (Custom-developed)

## Challenges we ran into

We restarted 9 times! Most of which revolved around integrating CORS with the front end react. Processing data in the backend while trying to fix these errors pushed us back several time.
Being able to set up a data structure and being able to combine all the features together so they are able to automatically and seamlessly.

## Accomplishments that we're proud of

### Creating of Ruff

Ruff is a smart AI that evolves with the software. As it processes more data, it detects patterns in patient histories, treatments, and outcomes, continuously improving its accuracy. Over time, Ruff’s success rate increases, offering better recommendations for medications and treatments.

Ruff adapts to new data, making it more effective at predicting successful outcomes, enhancing patient care, and streamlining decision-making for doctors. The more Ruff is used, the smarter and more reliable it becomes, providing valuable, data-driven insights to healthcare professionals.

### Being a team of 2

By being a team of 2 and being first time hackers it was a lot of pressure, however, we walked in with a goal on our minds and we knew each other’s skills that helped us innovate better, for the whole hackathon we had around 2-3 hours of sleep. But as we passed through multiple hours of coding it seemed just as another programming project.
Proper planning and time management allowed us to complete our project parts on time.

## What we learned

We learned to think quickly and rely on each other, understanding that if one person makes a mistake, it affects the whole team. This taught us the importance of collaboration and accountability, reinforcing the idea that success is a shared responsibility.

Additionally, we gained valuable skills in organizing and structuring complex databases, ensuring they were efficient and scalable. We also dove into the process of creating AI, learning not just how to build these intelligent systems but also understanding the underlying mechanics of how AI learns, adapts, and improves over time.

## What's next for ConnectHealth

### 1. **Advanced AI for Ruff**

* **NLP**: Enhance Ruff’s understanding of complex medical commands for better interaction.
* **IoMT (Internet of Medical Things)**: By connecting Ruff to IoMT, it will have access to real-time, up-to-date data, significantly enhancing its ability to process information.

### 2. **Mobile App**

* **Cross-Platform Access**: Build an iOS/Android app for easy access to patient data.
* **Push Notifications**: Remind patients of appointments or prescription refills.

### 3. **Interoperability**

* **EHR Integration**: Connect with other EHR systems for seamless data sharing between hospitals.
* **Pharmacy Integration**: Streamline prescription ordering with direct connections to pharmacies.