## What Is It

The Air Synth is a virtual synthesizer that can be played without the need for a physical instrument. By simply moving your fingers in the air, the Air Synth matches your motions to the correct pitches and allows you to practice, jam, and compose wherever and whenever you want.

## How It Works

The Air Synth uses OpenCV to detect the contours of a hand. This is done with background subtraction, a technique that compares a static background image with the live camera feed. The resulting image is passed through a series of filters: black and white filter and a gaussian blur. Then the contour lines are drawn over the hand and critical points are identified. We map these critical points to a range of y-values on the GUI in order to determine which note should be played.
## Inspiration

Many people were left unprepared to effectively express their ideas over Zoom. As students we’ve noticed that with remote learning, instructors seem to feel very distant, teaching through either slides or drawing on a tablet, without us being able to see their face. We know body language is an important part of being a presenter, so why should Zoom limit us like this? Aside from instructional content, Zoom meetings are boring so we ought to spice it up. Introducing AirCanvas!

## What it does

AirCanvas allows you to have a virtual whiteboard powered by Deep Learning and Computer Vision! Simply stick out your index finger as the pen, and let your imagination unfold in the air that is your canvas. This adds an entirely new dimension of communication that is not possible in person. You can also erase the drawings, drag the canvas around and draw with both hands simultaneously!

## How we built it

We begin with OpenPose, a project by researchers at Carnegie Mellon which performs pose estimation on a video stream in real time. With it, we are able to extract information about one’s poses, including the locations of joints. With this information, we have developed our own algorithm to recognize gestures that indicate the user’s desired actions. We additionally employ motion smoothing, which we will discuss a bit later. Combining these, we can render the user’s drawings onto the webcam stream, as if the user is drawing in mid-air.

## Challenges we ran into

One of the major challenges was, due to the low framerates of OpenPose, the paths generated are choppy, where the lines go back and forth. To solve this issue, we employed motion smoothing, and we can do this rather efficiently in O(1) runtime with our own data structure. This way, we get smooth lines without much performance penalty. In addition, we spent many hours testing our model and refining hyperparameters to ensure it works intuitively.

## What's next for AirCanvas

We were inexperienced with this technology, and we wish we had more time to add the following features:
Support for multiple people drawing with one camera simultaneously
A friendlier interface for user to customize pen settings such as color, thickness etc.
Tools you’d see in note taking apps, such as ruler
## Inspiration

A deep and unreasonable love of xylophones

## What it does

An air xylophone right in your browser!
Play such classic songs as twinkle twinkle little star, ba ba rainbow sheep and the alphabet song or come up with the next club banger in free play.

We also added an air guitar mode where you can play any classic 4 chord song such as Wonderwall

## How we built it

We built a static website using React which utilised Posenet from TensorflowJS to track the users hand positions and translate these to specific xylophone keys.

We then extended this by creating Xylophone Hero, a fun game that lets you play your favourite tunes without requiring any physical instruments.

## Challenges we ran into

Fine tuning the machine learning model to provide a good balance of speed and accuracy

## Accomplishments that we're proud of

I can get 100% on Never Gonna Give You Up on XylophoneHero (I've practised since the video)

## What we learned

We learnt about fine tuning neural nets to achieve maximum performance for real time rendering in the browser.

## What's next for XylophoneHero

We would like to:

* Add further instruments including a ~~guitar~~ and drum set in both freeplay and hero modes
* Allow for dynamic tuning of Posenet based on individual hardware configurations
* Add new and exciting songs to Xylophone
* Add a multiplayer jam mode