## Inspiration

During modern times, the idea underlying a facemask is simple--if more people wear them, less people will get sick. And while it holds true, this is an oversimplification: the number of lives saved is dependent not only on the quantity, but also on the quality of the masks which people wear (as evidenced by recent research by the CDC). However, due to an insufficient supply of N95 masks, healthcare workers are forced to wear cloth or surgical masks which both leak from the sides, increasing the risk of infection, and are arduous to breathe through for extended physical exertion.

## What it does

Maskus is the first mask bracket and fitter in one - custom-fitted and printed using accessible technology. It is designed to improve the baseline quality of facemasks around the world, with its first and most pressing use is for healthcare workers. The user starts taking a picture of their face through their computer/smartphone camera. We then generate an accurate 3D representation of the user's face and design a tight-fitting 3D printable mask bracket specifically tailored to the user's face contours. Within seconds, we can render the user's custom mask onto the user's face in augmented reality in realtime. The user can then either download their custom mask in a format ready for 3D printing, or set up software to print the mask automatically. We also have an Arduino Nano that alerts the user if the mask is secured properly, or letting them now it needs to be readjusted.

## How we built it

After the user visits the Maskus website, our React frontend sends a POST request to a Python Flask backend. The server receives the image, decodes it, and feeds it into a state of the art machine learning 3D face reconstruction model (3DDFA). The resultant 3D face model then goes through some preprocessing, which compresses the 3D data to improve performance. Another script then extracts the user's face contour/outline from the 3D model and builds a custom mask bracket with programmable CAD software. On the web app, the user gets to see both their own 3D face mesh as well as an AR rendering of the custom fitted mask onto their face (using React and three.js). Lastly, this data is saved to a standard 3D printing file format (.obj) and returned to the user so they can print it wherever they like. In terms of our hardware, the mask's alert system comprises of an Arduino Nano with a piezo buzzer and two push buttons (left and right side of face) wired in series. In order to get the push buttons to engage when the mask is worn, we created custom 3D parts that create a larger area for the buttons to be pushed.

## Challenges we ran into

This project was touched many disciplines, and posed many difficulties. We were determined to provide the user with the ability to see how their mask would fit them in real time using AR. In order to do this, we needed a way to visualize 3D models in the web. This proved difficult due to many misleading resources and weak documentation. Simple things (like figuring out how to get a 3D model to stop rotating) took much longer than they should have, simply because the frameworks were obfuscated. AR was also very difficult to implement, particularly due to the fact that it is a new technology and the existing frameworks for it are not yet mature. Our project is one of the first we've seen placing 3D models (not images) onto user faces.

## Accomplishments that we're proud of

From the machine learning side of the project, 3D face reconstruction is a very difficult problem. Luckily, our team was able to succesfully implement and use the 3DDFA state of the art machine learning model for face reconstruction. Installing and configuring the neccessary Python packages and virtual environments posed a challenge at the start, but we were able to quickly overcome this and get a working machine learning pipeline. Being able to solve this problem early on in the hackathon gave our team more time to focus on other problems, such as web 3D model visualization and constructing the facemask from our 3D face model.

## What we learned

Amusingly, during this project we found that things which were supposed to be difficult turned out to be easy to implement and, conversely, the easy parts turned out to be hard. Things like front end design and integrating web frameworks turned out to be some of the most challenging parts of the project, whereas things like machine learning were easier than expected. A takeaway is that the feasibility of quickly building a project should be based not only on the difficulty of the task, but also on the quality of existing resources which can be used to build it. Good frameworks make implementing difficult projects much easier.

## What's next for Maskus

Aside from refactoring the code and improving webpage design, we see several things for the project going forward. Perhaps the biggest points is developing a reliable algorithm to extract the facemask outline from a 3D face model. The one the group currently has works most of the time, but serves as the bottleneck of the system in terms of facial recognition accuracy. The UI design can be improved as well. Lastly, threeJS was found to be a pain, especially when trying to integrate it with React. It would be worth exploring simpler JavaScript frameworks. We would also love to add more functionality to the Arduino in the future, making it a 'smarter' mask. We hope to add sensors like AQS (Air Quality Sensor), creating alerts if the mask has been worn too long and needs to be replaced, and status LEDs in order to visually tell your mask is secure.

In terms of future growth, Markus can comfortably be deployed as a web app and used by healthcare workers around the world in order to decrease risk of COVID transmission. It is a low cost solution designed to work with existing masks and improve upon them. Opening up the software to open source contribution is a potential way to grow, and we hope it would lead to very fast progress.
## Inspiration

We all look forward to being home. The familiarity of snuggling up in our comfortable beds and the feeling of our own shower pressure is what we desire - even after a luxurious vacation. But when faced with the cold and dark of an empty house, the word "home" almost doesn't fit. Haven aims to make the event of opening the front door a pleasurable one. We hope to redefine the meaning of "welcome home" by personalizing and enhancing the experience.

While there are numerous smart home technologies developed in the real world, we aim to create a new culture along with new technologies. The consumer doesn't value taking the complicated steps to set up a smart home, and in addition are heavily worried about security and privacy. Though we can't get people to trust us, we rely on people trusting each other to create the smart home norm.

## What it does

Haven is not just an app or a function. It is a platform. Haven uses machine learning and image processing to recognize faces that are registered in the household. It then adjusts features in the house such as unlocking the door, adjusting lighting, heat, and anything else the user might want to create the most welcoming experience. It leaves behind the days of coming home to a lonely, stark house after a long tiring day. The API allows easy user registration and security, and the broad scope of our technology allows more and more functions to be added in the future, creating the most personalized experience possible.

We added a functions library similar to the smartphone's app store. This allows a public run development platform made for the public. With simpler steps. come more users, with more users, comes more trust, and with more trust, comes the popularization of smart homes.

## How we built it

Haven was built using Python and Flask for the face recognition and API respectively. It is hosted using Amazon AWS and utilises libraries including OpenCV for image processing and OpenFace for face recognition.

The hardware was powered a QualComm DragonBoard, which utilized a USB camera to take continuous images, which were processed by cv and sent to our server. This server powered our face recognition, utilizing OpenFace neural networks that we trained from many pictures of our own face. Tagged by an ID, the board receives the necessary permissions to trigger all of its actions. The homeowner's face can then turn on a light, or open the door.

Then, on the front end, we wire-framed our user story on paper before proceeding to create graphics on Adobe Illustrator and screens on Sketch. They were then imported into Principle to create the guiding animations and finally to create the user-centred software platform and to our smart technology.

## Challenges we ran into

Since no one on our team had any experience with machine learning and neural networks, we had to learn a lot very quickly in order to implement face recognition. Learning to use these complex algorithms were difficult, completely understanding them would have needed a Phd.

Hosting our application on AWS turned out to be difficult because setting up OpenFace without Docker had a lot of set-up and required a lot of storage; requiring us to attempt to set-up our servers multiple times.

On the hardware side of things, we found that we struggled the most with performing the necessary functions with the QualComm Dragonboard. Documentation was sparse and as a result we struggled quite a bit with necessary permissions and dealing with dependencies.

## Accomplishments that we're proud of

Although we are nowhere close to being experts on this subject, it was definitely super cool to see how image processing, even given a slightly limited set of data, could be so helpful and accurate in building our solution. Hosting this type of computation over the web was certainly not easy either, and we're happy about how well that turned out, as well as how seamless the integrations became as a result of AWS.

On the user experience front, we feel accomplished about the seamless animated interactions integrated into the front-end of our software platform. The custom illustrations, transitions, and user profiles allow an easy visual understanding of the Haven technology.

## What we learned

For some of our team members, it was our first hackathons. Learning to work as a very key subunit to a bigger team in a development setting was a new experience, especially seeing pieces fit together in the end. Some of us have never worked with Git.

By utilising neural networks and machine learning without having prior knowledge, we learned a lot about how neural networks and machine learning works. We've always joked about these "buzzwords", but when it comes to working with it, it certainly isn't easy and will always remind us how much there is to learn still.

## What's next for Haven

In order to truly give each individual their own home welcoming, we hope that there will be growth in potential functions that Haven can execute. Though we are providing the technology, security, and platform. Our vision is for the functions to be written by our users, for our users, just like skill on the Amazon Alexa, or apps for iOS to wholeheartedly capture customization and popularization of the smart home.
## Inspiration

When visiting a clinic, two big complaints that we have are the long wait times and the necessity to use a kiosk that thousands of other people have already touched. We also know that certain methods of filling in information are not accessible to everyone (For example, someone with Parkinsons disease writing with a pen). In response to these problems, we created Touchless.

## What it does

* Touchless is an accessible and contact-free solution for gathering form information.
* Allows users to interact with forms using voices and touchless gestures.
* Users use different gestures to answer different questions.
* Ex. Raise 1-5 fingers for 1-5 inputs, or thumbs up and down for yes and no.
* Additionally, users are able to use voice for two-way interaction with the form. Either way, surface contact is eliminated.
* Applicable to doctor’s offices and clinics where germs are easily transferable and dangerous when people touch the same electronic devices.

## How we built it

* Gesture and voice components are written in Python.
* The gesture component uses OpenCV and Mediapipe to map out hand joint positions, where calculations could be done to determine hand symbols.
* SpeechRecognition recognizes user speech
* The form outputs audio back to the user by using pyttsx3 for text-to-speech, and beepy for alert noises.
* We use AWS Gateway to open a connection to a custom lambda function which has been assigned roles using AWS Iam Roles to restrict access. The lambda generates a secure key which it sends with the data from our form that has been routed using Flask, to our noSQL dynmaoDB database.

## Challenges we ran into

* Tried to set up a Cerner API for FHIR data, but had difficulty setting it up.
* As a result, we had to pivot towards using a noSQL database in AWS as our secure backend database for storing our patient data.

## Accomplishments we’re proud of

This was our whole team’s first time using gesture recognition and voice recognition, so it was an amazing learning experience for us. We’re proud that we managed to implement these features within our project at a level we consider effective.

## What we learned

We learned that FHIR is complicated. We ended up building a custom data workflow that was based on FHIR models we found online, but due to time constraints we did not implement certain headers and keys that make up industrial FHIR data objects.

## What’s next for Touchless

In the future, we would like to integrate the voice and gesture components more seamlessly into one rather than two separate components.