## Inspiration

Have you ever tried learning the piano but never had the time to do so? Do you want to play your favorite songs right away without any prior experience? This is the app for you. We required something like this even for our personal use. It helps an immense lot in building muscle memory to play the piano for any song you can find on Youtube directly.

## What it does

Use AR to project your favorite digitally-recorded piano video song from Youtube over your piano to learn by following along.

## Category

Best Practicable/Scalable, Music Tech

## How we built it

Using Unity, C# and the EasyAR SDK for Augmented Reality

## Challenges we ran into

* Some Youtube URLs had cipher signatures and could not be loaded asynchronously directly
* App crashes before submission, fixed barely on time

## Accomplishments that we're proud of

We built a fully functioning and user friendly application that perfectly maps the AR video onto your piano, with perfect calibration. It turned out to be a lot better than we expected. We can use it for our own personal use to learn and master the piano.

## What we learned

* Creating an AR app from scratch in Unity with no prior experience
* Handling asynchronously loading of Youtube videos and using YoutubeExplode
* Different material properties, video cipher signatures and various new components!
* Developing a touch gesture based control system for calibration

## What's next for PianoTunesAR

* Tunes List to save the songs you have played before by name into a playlist so that you do not need to copy URLs every time
* Machine learning based AR projection onto piano with automatic calibration and marker-less spatial/world mapping
* All supported Youtube URLs as well as an Upload your own Video feature
* AR Cam tutorial UI and icons/UI improvements
* iOS version

# APK Releases

<https://github.com/hamzafarooq009/PianoTunesAR-Wallifornia-Hackathon/releases/tag/APK>
## Inspiration

Every musician knows that moment of confusion, that painful silence as onlookers shuffle awkward as you frantically turn the page of the sheet music in front of you. While large solo performances may have people in charge of turning pages, for larger scale ensemble works this obviously proves impractical. At this hackathon, inspired by the discussion around technology and music at the keynote speech, we wanted to develop a tool that could aid musicians.

Seeing AdHawks's MindLink demoed at the sponsor booths, ultimately give us a clear vision for our hack. MindLink, a deceptively ordinary looking pair of glasses, has the ability to track the user's gaze in three dimensions, recognizes events such as blinks and even has an external camera to display the user's view. Blown away by the possibility and opportunities this device offered, we set out to build a hands-free sheet music tool that simplifies working with digital sheet music.

## What it does

Noteation is a powerful sheet music reader and annotator. All the musician needs to do is to upload a pdf of the piece they plan to play. Noteation then displays the first page of the music and waits for eye commands to turn to the next page, providing a simple, efficient and most importantly stress-free experience for the musician as they practice and perform. Noteation also enables users to annotate on the sheet music, just as they would on printed sheet music and there are touch controls that allow the user to select, draw, scroll and flip as they please.

## How we built it

Noteation is a web app built using React and Typescript. Interfacing with the MindLink hardware was done on Python using AdHawk's SDK with Flask and CockroachDB to link the frontend with the backend.

## Challenges we ran into

One challenge we came across was deciding how to optimally allow the user to turn page using eye gestures. We tried building regression-based models using the eye-gaze data stream to predict when to turn the page and built applications using Qt to study the effectiveness of these methods. Ultimately, we decided to turn the page using right and left wink commands as this was the most reliable technique that also preserved the musicians' autonomy, allowing them to flip back and forth as needed.

Strategizing how to structure the communication between the front and backend was also a challenging problem to work on as it is important that there is low latency between receiving a command and turning the page. Our solution using Flask and CockroachDB provided us with a streamlined and efficient way to communicate the data stream as well as providing detailed logs of all events.

## Accomplishments that we're proud of

We're so proud we managed to build a functioning tool that we, for certain, believe is super useful. As musicians this is something that we've legitimately thought would be useful in the past, and granted access to pioneering technology to make that happen was super exciting. All while working with a piece of cutting-edge hardware technology that we had zero experience in using before this weekend.

## What we learned

One of the most important things we learnt this weekend were the best practices to use when collaborating on project in a time crunch. We also learnt to trust each other to deliver on our sub-tasks and helped where we could. The most exciting thing that we learnt while learning to use these cool technologies, is that the opportunities are endless in tech and the impact, limitless.

## What's next for Noteation: Music made Intuitive

Some immediate features we would like to add to Noteation is to enable users to save the pdf with their annotations and add landscape mode where the pages can be displayed two at a time. We would also really like to explore more features of MindLink and allow users to customize their control gestures. There's even possibility of expanding the feature set beyond just changing pages, especially for non-classical musicians who might have other electronic devices to potentially control. The possibilities really are endless and are super exciting to think about!
## Inspiration

We really enjoy Virtual Reality and wanted to work with the Oculus DK2. Combining this with our love for music, we thought it would be interesting and fun to find a way to visualize music in a three - dimensional context.

## What it does

HearVR is a program that combines Machine Learning with Virtual Reality to create an environment where SoundCloud music files can be played and visualized through a frequency spectrum and user - created comments. Music files and corresponding comments are gathered from SoundCloud and a sentiment analysis is performed on the comments. The music files are then played in a virtual three - dimensional environment where each song has a corresponding frequency spectrum and a stream of comments that are color - coordinated to represent how positive or negative it is. The user can traverse the virtual space to explore different songs.

## How we built it

We wrote a Python script that uses multiprocessing to concurrently download and process SoundCloud files as it retrieves SoundCloud comments and communicates with an Azure web service that runs a sentiment analysis on the comments and gives it a score based on how positive or negative the comment is. Each music file's comments, the comments' scores, and additional information about the comments are written in a Comma Separated Value (CSV) file. The CSV file for each song is accessed by Unity, which uses C# with Visual Studio as an IDE to parse the CSV and create the corresponding frequency spectrum and comments. We then designed the virtual environment in Unity to produce a visual layout that displays the frequency spectrum and streams the comments according to their time stamp on SoundCloud for each song.

## Challenges we ran into

Since we were working with VR, the technology we were using was very immature. We initially started out with Unreal Engine for our project - however, we quickly found that Unreal's audio engine was buggy and unreliable. After too many hours, we switched to Unity, a tool which none of us had worked with. Unity was a huge learning curve, but we pushed through. However, we had more issues - Unity can't decode MP3's and our plan to stream from SoundCloud was over. Instead, we did some trickery using Python to preprocess the MP3's into WAV files before feeding them to Unity. On the backend, we struggled through Microsoft Azure, which was also new technology to us.

## Accomplishments that we're proud of

We're really proud of combining our interests in Machine Learning and Virtual Reality together to create a unique program that enhances user experience in listening to music.

## What we learned

We learned new technologies such as Microsoft Azure, Unity, and how to integrate these different technologies together. Additionally, we learned a new language in C#.

## What's next for HearVR

We would like to make HearVR a world that generates music automatically based on the person's preferences and which songs they listened to. In addition, we would like to make this a networked experience, so multiple people can listen in on the same session