## Bringing your music to life, not just to your ears but to your eyes 🎶

## Inspiration 🍐

Composing music through scribbling notes or drag-and-dropping from MuseScore couldn't be more tedious. As pianists ourselves, we know the struggle of trying to bring our impromptu improvisation sessions to life without forgetting what we just played or having to record ourselves and write out the notes one by one.

## What it does 🎹

Introducing PearPiano, a cute little pear that helps you pair the notes to your thoughts. As a musician's best friend, Pear guides pianists through an augmented simulation of a piano where played notes are directly translated into a recording and stored for future use. Pear can read both single notes and chords played on the virtual piano, allowing playback of your music with cascading tiles for full immersion. Seek musical guidance from Pear by asking, "What is the key signature of C-major?" or "Tell me the notes of the E-major diminished 7th chord." To fine tune your compositions, use "Edit mode," where musicians can rewind the clip and drag-and-drop notes for instant changes.

## How we built it 🔧

Using Unity Game Engine and the Oculus Quest, musicians can airplay their music on an augmented piano for real-time music composition. We used OpenAI's Whisper for voice dictation and C# for all game-development scripts. The AR environment is entirely designed and generated using the Unity UI Toolkit, allowing our engineers to realize an immersive yet functional musical corner.

## Challenges we ran into 🏁

* Calibrating and configuring hand tracking on the Oculus Quest
* Reducing positional offset when making contact with the virtual piano keys
* Building the piano in Unity: setting the pitch of the notes and being able to play multiple at once

## Accomplishments that we're proud of 🌟

* Bringing a scaled **AR piano** to life with close-to-perfect functionalities
* Working with OpenAI to synthesize text from speech to provide guidance for users
* Designing an interactive and aesthetic UI/UX with cascading tiles upon recording playback

## What we learned 📖

* Designing and implementing our character/piano/interface in 3D
* Emily had 5 cups of coffee in half a day and is somehow alive

## What's next for PearPiano 📈

* VR overlay feature to attach the augmented piano to a real one, enriching each practice or composition session
* A rhythm checker to support an aspiring pianist to stay on-beat and in-tune
* A smart chord suggester to streamline harmonization and enhance the composition process
* Depth detection for each note-press to provide feedback on the pianist's musical dynamics
* With the up-coming release of Apple Vision Pro and Meta Quest 3, full colour AR pass-through will be more accessible than ever — Pear piano will "pair" great with all those headsets!
## Inspiration

Pianos are usually heavy and expensive, so a keyboard made from cheap and portable materials like paper can be very useful. That's why we came up with the paper piano idea. With a paper piano, it's natural to connect it to a computer and achieve multiple purposes like composing. While most composers rely on midi file export to view and make adjustment to their compositions, we wish to realize real-time visual representation of the music being played so people can view their music notes in real time.

## What it does

Pianeer is a highly innovated and convenient music composing software+hardware system. Multiple modes are available for different musical functions. It includes a paper piano which provides real experience of playing keyboard but can be easily rolled up and carried. The composing mode of the software provides real time visual representation of the music that's being played and exports a midi file. The play mode provides sheet music and keyboard highlights for practicing purpose. The practice mode allows users to practice their compositions casually without being recorded. Pianeer is extremely multi-functional and portable, and can be used for both beginners and mature composers.

## How we built it

We put electric paint on a hard paper and connect it to arduinos via wires. Using capacitive sensing library we are able to give input signals by simply putting fingers on the paper. We exported the input signal to our program to generate sound and realize other functions.

## Challenges we ran into

* It's very hard to accurately draw so many keys on the paper using electric paint since each key has to be clearly separated.
* Adjusting delay and accuracy of the paper piano is very challenging.
* Convert music input signals to visual representation.
* User Interface design.

## Accomplishments that we're proud of

* Lovely user interface that corresponds to the superhero theme.
* Pretty accurate sensing and sound generation of the paper piano.
* Real time visual representation of the music
* Various functions and modes we've accomplished.

## What we learned

Pygame, arduino, multi-threading

## What's next for Pianeer

Make a more delicate prototype of the paper piano and realize the player mode we have no time to accomplish this time.
## 💡 Inspiration 💡

Some of our team members have played musical instruments in the past. They know the pain of stopping or pausing while playing the music to flip over their music sheet. Using the hands to flip over a music sheet is not ideal since the hands should be on the instrument to play the music at all times. Since the eyes have to be on the music sheet to read music, why not use the eyes to flip over the music pages instead? And keep our hands on the instrument.

## ❓ What it does ❓

TuneTurn tracks the angle of the musician's head using openCV and uses it to detect whether the user's head is turned to the left or right, and accordingly flips the page of the music sheets, allowing for a hands-free experience. The app also has a vast library of music stored which the user can access and listen to and get a better idea of how to actually play the song of their choice before they start.

## 🤔 How we built it 🤔

Using computer vision (OpenCV) in an app deployed using Qualcomm HDK 8450, we track the movement of the head to trigger the change of page based on which way the head is moving based on the angle that is formed. We ensure then send a request to a Flask server, and ensure only 1 request is sent even though the user's head will be in the turned position for a few hundred milliseconds. The server processes the request and accordingly flips the sheet by using pyautogui to control the computer's left/right keys.

## 😰 Challenges we ran into 😰

It was our first time using OpenCV and we struggled a lot with accurately tracking the coordinates of the desired landmarks on the user's face as choosing the wrong ones would make it difficult to accurately detect whether the user's head is turned. Another major challenge was using Android App Studio to do the ML using Qualcomm as we had never used it before. We primarily code in Python and found the drastic change to Java quite difficult, especially since we had to do relatively complex ML using openCV.

## 🥇 Accomplishments that we're proud of 🥇

We are very proud off making an accurate openCV model by choosing good landmarks and calculating the angle between them to detect whether, and in what direction the user's head is turned. We are also proud of using very different platforms for our application and making them interact with each other and also using a completely different device to host our app, the Qualcomm HDK 8450, instead of a laptop which is what we are used to.

## 🎼 What we learned 🎼

We learned how to make effective mobile apps using Android App Studio and using completely different hardware to power our app. We took a large step out of our comfort zone and experimented with new technologies, allowing us to greatly progress in a short time. We also distributed the tasks accurately to ensure our workflow was streamlined.

## 😮 What's next for MakeUofT 😮

We are planning to make our feature which converts music to notes of a music sheet more accurate so we can better compare the user's music with the original music. To do this, we might try to make our own model and train it, and if it proves to be too hard we will try to find a better API.