## Bringing your music to life, not just to your ears but to your eyes 🎶

## Inspiration 🍐

Composing music through scribbling notes or drag-and-dropping from MuseScore couldn't be more tedious. As pianists ourselves, we know the struggle of trying to bring our impromptu improvisation sessions to life without forgetting what we just played or having to record ourselves and write out the notes one by one.

## What it does 🎹

Introducing PearPiano, a cute little pear that helps you pair the notes to your thoughts. As a musician's best friend, Pear guides pianists through an augmented simulation of a piano where played notes are directly translated into a recording and stored for future use. Pear can read both single notes and chords played on the virtual piano, allowing playback of your music with cascading tiles for full immersion. Seek musical guidance from Pear by asking, "What is the key signature of C-major?" or "Tell me the notes of the E-major diminished 7th chord." To fine tune your compositions, use "Edit mode," where musicians can rewind the clip and drag-and-drop notes for instant changes.

## How we built it 🔧

Using Unity Game Engine and the Oculus Quest, musicians can airplay their music on an augmented piano for real-time music composition. We used OpenAI's Whisper for voice dictation and C# for all game-development scripts. The AR environment is entirely designed and generated using the Unity UI Toolkit, allowing our engineers to realize an immersive yet functional musical corner.

## Challenges we ran into 🏁

* Calibrating and configuring hand tracking on the Oculus Quest
* Reducing positional offset when making contact with the virtual piano keys
* Building the piano in Unity: setting the pitch of the notes and being able to play multiple at once

## Accomplishments that we're proud of 🌟

* Bringing a scaled **AR piano** to life with close-to-perfect functionalities
* Working with OpenAI to synthesize text from speech to provide guidance for users
* Designing an interactive and aesthetic UI/UX with cascading tiles upon recording playback

## What we learned 📖

* Designing and implementing our character/piano/interface in 3D
* Emily had 5 cups of coffee in half a day and is somehow alive

## What's next for PearPiano 📈

* VR overlay feature to attach the augmented piano to a real one, enriching each practice or composition session
* A rhythm checker to support an aspiring pianist to stay on-beat and in-tune
* A smart chord suggester to streamline harmonization and enhance the composition process
* Depth detection for each note-press to provide feedback on the pianist's musical dynamics
* With the up-coming release of Apple Vision Pro and Meta Quest 3, full colour AR pass-through will be more accessible than ever — Pear piano will "pair" great with all those headsets!
## Inspiration

Every musician knows that moment of confusion, that painful silence as onlookers shuffle awkward as you frantically turn the page of the sheet music in front of you. While large solo performances may have people in charge of turning pages, for larger scale ensemble works this obviously proves impractical. At this hackathon, inspired by the discussion around technology and music at the keynote speech, we wanted to develop a tool that could aid musicians.

Seeing AdHawks's MindLink demoed at the sponsor booths, ultimately give us a clear vision for our hack. MindLink, a deceptively ordinary looking pair of glasses, has the ability to track the user's gaze in three dimensions, recognizes events such as blinks and even has an external camera to display the user's view. Blown away by the possibility and opportunities this device offered, we set out to build a hands-free sheet music tool that simplifies working with digital sheet music.

## What it does

Noteation is a powerful sheet music reader and annotator. All the musician needs to do is to upload a pdf of the piece they plan to play. Noteation then displays the first page of the music and waits for eye commands to turn to the next page, providing a simple, efficient and most importantly stress-free experience for the musician as they practice and perform. Noteation also enables users to annotate on the sheet music, just as they would on printed sheet music and there are touch controls that allow the user to select, draw, scroll and flip as they please.

## How we built it

Noteation is a web app built using React and Typescript. Interfacing with the MindLink hardware was done on Python using AdHawk's SDK with Flask and CockroachDB to link the frontend with the backend.

## Challenges we ran into

One challenge we came across was deciding how to optimally allow the user to turn page using eye gestures. We tried building regression-based models using the eye-gaze data stream to predict when to turn the page and built applications using Qt to study the effectiveness of these methods. Ultimately, we decided to turn the page using right and left wink commands as this was the most reliable technique that also preserved the musicians' autonomy, allowing them to flip back and forth as needed.

Strategizing how to structure the communication between the front and backend was also a challenging problem to work on as it is important that there is low latency between receiving a command and turning the page. Our solution using Flask and CockroachDB provided us with a streamlined and efficient way to communicate the data stream as well as providing detailed logs of all events.

## Accomplishments that we're proud of

We're so proud we managed to build a functioning tool that we, for certain, believe is super useful. As musicians this is something that we've legitimately thought would be useful in the past, and granted access to pioneering technology to make that happen was super exciting. All while working with a piece of cutting-edge hardware technology that we had zero experience in using before this weekend.

## What we learned

One of the most important things we learnt this weekend were the best practices to use when collaborating on project in a time crunch. We also learnt to trust each other to deliver on our sub-tasks and helped where we could. The most exciting thing that we learnt while learning to use these cool technologies, is that the opportunities are endless in tech and the impact, limitless.

## What's next for Noteation: Music made Intuitive

Some immediate features we would like to add to Noteation is to enable users to save the pdf with their annotations and add landscape mode where the pages can be displayed two at a time. We would also really like to explore more features of MindLink and allow users to customize their control gestures. There's even possibility of expanding the feature set beyond just changing pages, especially for non-classical musicians who might have other electronic devices to potentially control. The possibilities really are endless and are super exciting to think about!
## Inspiration

I hate reading through job descriptions and trying to figure out what skills they are looking for in a developer amidst the shameless self-promotion and flagrant buzz-word spamming. So I decided to train a custom AI model to do so, and ended up building a whole AI training UI in the process!

## What it does

Nina allows a user to generate training code for custom MITIE models. It is project-agnositic, meaning it can be used to generate code for any kind of MITIE NER custom model.

Lumbergh is a custom trained MITIE model which parses through a job description and pulls out the required developer skills.

## How I built it

I build Nina using HTML, CSS, and JS (jQuery + Vue.js)

For Lumbergh I used MITIE, a deep learning library that is designed for perfoming Named Entity Recognition. I used Nina to generate custom MITIE training code, and then trained a model based on data which I pulled from the Indeed.com API and tagged by hand

## Challenges I ran into

Initially the model I designed was far too complex, and lead to very long AI training times (the first training session took an hour and a half, and that was only for 8 data sets!). I then simplified the model to pull just
one entity instead of 3. This not only lowered the training time, but it ended up leading to a more accurate model

## Accomplishments that I'm proud of

The accomplishment I am most proud of is the F1-Score I obtained on the custom model after optimizing it. An F1 score is a measure of accuracy for models. The highest F1 score I achieved was 91.2% on my final model.

## What I learned

I had never used machine learning before this weekend, I learned a ton about how deep learning and NER work behind the scenes. I also learned Vue.js for the first time to build the front end.

## What's next for Nina & Lumbergh

The plan is to continue improving the accuracy of Lumbergh and eventually use it as a feature in another app of mine called [ResOptim](https://github.com/jaydenwindle/mchacks2017), which automatically optimizes resumes based on a set of skill criteria. Currently Lumbergh is only trained on a dataset of 25 job descriptions, I would like to push that up to 100.

I plan to release Nina as a standalone tool to make generating MITIE model code easier for everyone who is working on NER models, as there are currently no good GUIs for that process