## Inspiration

Some of our team members are hobbyist DJs, playing shows for college parties and other local events. As smaller-size DJs we've always dreamed of having amazing live visuals for our sets, at the same quality as the largest EDM festivals; however, the cost, effort, and skillset was clearly infeasible for us, and infeasible for any musicians but the higher-profile ones who have the budget to commission lengthy visualizations and animations from third-party digital artists. That was a few years ago; recently, since generative vision models in ML have reached such a high quality of output, and open-source models (namely, stable diffusion) allow for intricate, transformative engineering at the level of model internals, we've realized our music live visuals dream has become possible.

We've come up with an app that takes a song and a prompt, and generates a music video in real-time that's perfectly synced to the music. The app will allow DJs and other musicians to have professional-level visuals for their live shows without the need for expensive commissions or complex technical setups. With our app, any musician can have access to stunning visuals that will take their performances to the next level.

## What it does

Mixingjays is an innovative video editing tool powered by machine learning that's designed specifically for creating stunning visual accompaniments to music. Our app allows clients to upload their music tracks and specify various parameters such as textures, patterns, objects, colors, and effects for different sections of the song.

Our system then takes these specifications and translates them into prompts for our advanced ML stable diffusion model, which generates a unique video tailored to the uploaded music track. Once the video is generated, the user can easily edit it using our intuitive and user-friendly interface. Our app provides users with a range of editing options, from basic graphical edits (similar to popular photo editing apps like Instagram and VSCO) to advanced generative edits that are handled by our ML pipeline.

The user experience of Mixingjays is similar to that of well-known video editing software like Adobe Premiere or Final Cut, but with the added power of our machine learning technology driving the video outputs. Our app provides users with the ability to create professional-quality music videos with ease and without the need for extensive technical knowledge. Whether you're a professional musician or an amateur DJ, Mixingjays can help you create stunning visuals that will take your music to the next level while giving the musicians full creative control over the video.

## How we built it

One of our baseline innovations is generating video from stable diffusion and engineering on the model internals of stable diffusion. As a standalone, stable diffusion outputs static images based on natural language prompts. Given a list of images that stable diffusion outputs from several prompts, we're able to generate video between the images by interpolating between images in CLIP latent space. A detail of stable diffusion is that it actually computes its output on a latent space, a compressed coordinate system the the model abstractly uses to represent images and text; the latent vector in latent space is then decoded into the image we see. This latent space is well-behaved in Euclidean space, so we can generate new image outputs and smooth transitions between images, strung into a video, by linearly interpolating between the latent vectors of two different images before decoding every interpolated vector. If the two images we interpolate between are from two different prompts from the same seed of stable diffusion, or the same prompt across two different seeds of stable diffusion, the resulting interpolations and video end up being semantically coherent and appealing.[1] A nice augmentation to our interpolation technique is perturbing the latent vector in coordinate space, making a new latent vector anywhere in a small Lp-norm ball surrounding the original vector. (This resembles the setting of adversarial robustness in computer vision research, although our application is not adversarial.) As a result, given a list of stable diffusion images and their underlying latent vectors, we can generate video by latent-space-interpolating between the images in order.

We generate videos directly from the music, which relies on a suite of algorithms for music analysis. The demucs model from Facebook AI Research (FAIR) performs "stem separation," or isolation of certain individual instruments/elements in a music track; we pass our music track into the model and generate four new music tracks of the same length, containing only the vocals, drums, bass, and melodic/harmonic instruments of the track, respectively. With OpenAI Whisper, we're able to extract all lyrics from the isolated vocal track, as well as bucket lyrics into corresponding timestamps at regular intervals. With more classical music analysis algorithms, we're able to:

* convert our drum track and bass track into timestamps for the rhythm/groove of the track
* and convert our melody/harmony track into a timestamped chord progression, essentially the formal musical notation of harmonic information in the track.

These aspects of music analysis directly interface into our stable diffusion and latent-space-interpolation pipeline. In practice, the natural language prompts into stable diffusion usually consist of long lists of keywords specifying:

* objects, textures, themes, backgrounds
* physical details
* adjectives influencing tone of the picture
* photographic specs: lighting, shadows, colors, saturation/intensity, image quality ("hd", "4k", etc)
* artistic styles
Many of these keywords are exposed to the client at the UI level of our application, as small text inputs, sliders, knobs, dropdown menus, and more. As a result, the client retains many options for influencing the video output, independently of the music. These keywords are concatenated with natural language encapsulations of musical aspects:
* Lyrics are directly added to the prompts.
* Melodic/harmonic information and chord progressions map to different colors and themes, based on different chords' "feel" in music theory.
* Rhythm and groove is added to the interpolation system! We speed up, slow down, and alter our trajectory of movement though latent space in time with the rhythm.
The result is a high-quality visualizer that incorporates both the user's specifications/edits and the diverse analyzed aspects of the music track.

Theoretically, with good engineering, we'd be able to run our video generation and editing pipelines very fast, essentially in real time! Because interpolation occurs at the very last layer of the stable diffusion model internals, and video output from it depends only on a simple decoder instead of the entire stable diffusion model, video generation from interpolation is very fast and the runtime bottleneck depends only on stable diffusion. We generate one or two dozen videos from stable diffusion per video, so by generating each image in parallel on a wide GPU array, as well as incorporating stable diffusion speedups, we're able to generate the entire video for a music track in a few seconds!

## Challenges we ran into

Prompt generation: Generating good quality prompts is hard. We tried different ways of prompting the model to see its affect on the quality of generated images and output video and discovered we needed more description of the song in addition to the manually inputted prompt. This motivated us to look into music analysis.

Music Analysis: Our music analysis required pulling together a lot of disparate libraries and systems for digital signal processing that aren't typically used together, or popular enough to be highly supported. The challenge here was wrangling these libraries into a single system without the pipeline crumbling to obscure bugs.

Stable Diffusion: The main challenge in using stable diffusion for multiple image generation is the overhead cost, compute and time. Primitive implementations of stable diffusion took 30s to generate an image which made things hard for us since we need to generate multiple images against each song and prompt pair. Modal workshop at Treehacks was very helpful for us to navigate this issue. We used modal containers and stubs to bake our functions into the images such that they are run only when the functions are called and free the GPUs otherwise for other functions to run. It also helped us parallelise our code which made things faster. However, since it is new, it was difficult to get it to recognise other files, adding data types to function methods etc.

Interpolation: This was by far the hardest part. Interpolation is not only slow but also hard to implement or reproduce. After extensive research and trials with different libraries, we used Google Research’s Frame-interpolation method. It is implemented in tensorflow, in contrary to the other code which was more PyTorch heavy. In addition to that it is slow and scales exponentially with the number of images. A 1 minute video took 10 minutes to generate. Given the 36 hour time limitation, we had to generate the videos in advance for our app but there is tons of scope to make it faster.

Integration: The biggest bottlenecks came about when trying to generate visuals for arbitrary input mp3 files. Because of this, we had to reduce our expectation of on-the-fly, real-time rendering of any input file, to a set of input files that we generated offline. Render times ranged anywhere from 20 seconds to many minutes, which means that we’re still a little bit removed from bringing this forward as a real-application for users around the world to toy with. Real-time video editing capabilities also proved difficult to implement given the lack of time for building substantial infrastructure, but we see this as a next step in building out a fully functional product that anyone can use!

## Accomplishments that we're proud of

We believe we've made significant progress towards a finalized, professional-grade application for our purposes. We showed that the interpolation idea produces video output that is high-quality, semantically coherent, granularly editable, and closely linked to the concurrent musical aspects of rhythm, melody, harmony, and lyrics. This makes our video offerings genuinely usable in professional live performances. Our hackathon result is a more rudimentary proof-of-concept, but our results lead us to believe that continuing this project in a longer-term setting would lead to a robust, fully-featured offering that directly plugs into the creative workflow of hobbyist and professional DJs and other musical live performers.

## What we learned

We found ourselves learning a ton about the challenges of implementing cutting-edge AI/ML techniques in the context of a novel product. Given the diverse set of experiences that each of our team members have, we also learned a ton through cross-team collaboration between scientists, engineers, and designers.

## What's next for Mixingjays

Moving forward, we have several exciting plans for improving and expanding our app's capabilities. Here are a few of our top priorities:

Improving speed and latency: We're committed to making our app as fast and responsive as possible. That means improving the speed and latency of our image-to-video interpolation process, as well as optimizing other parts of the app for speed and reliability.

Video editing: We want to give our users more control over their videos. To that end, we're developing an interactive video editing tool that will allow users to regenerate portions of their videos and choose which generation they want to keep. Additionally, we're exploring the use of computer vision to enable users to select, change, and modify objects in their generated videos.

Script-based visualization: We see a lot of potential in using our app to create visualizations based on scripts, particularly for plays and music videos. By providing a script, our app could generate visualizations that offer guidance on stage direction, camera angles, lighting, and other creative elements. This would be a powerful tool for creators looking to visualize their ideas in a quick and efficient manner.

Overall, we're excited to continue developing and refining our app to meet the needs of a wide range of creators and musicians. With our commitment to innovation and our focus on user experience, we're confident that we can create a truly powerful and unique tool for music video creation.

## References

Seth Forsgren and Hayk, Martiros, "Riffusion - Stable diffusion for real-time music generation." 2022. <https://riffusion.com/about>
## Inspiration

Music is something inherently personal to each and every one of us - our favorite tracks accompany us through our highs and lows, through tough workouts and relaxing evenings. Our aim is to encourage and capture that feeling of discovering that new song you just can't stop listening to. Music is an authentic expression of ourselves, and the perfect way to [meet new people](https://www.lovethispic.com/uploaded_images/206094-When-You-Meet-Someone-With-The-Same-Music-Taste-As-You.jpg) without the clichés of the typical social media platforms we're all sick of. We're both very passionate about reviving the soul of social media, so we were very excited to hear about this track and work on this project!

## What it does

Spotify keeps tabs on the tracks you can't get enough of. Why not make that data work *for you*? With one simple login, ensemble matches you with others who share your musical ear. Using our *state-of-the-art* machine learning algorithms, we show you other users who we think you'd like based on both their and your music taste. Love their tracks? Follow them and stay tuned. ensemble is a new way to truly connect on a meaningful level in an age of countless unoriginal social media platforms.

## How we built it

We wanted a robust application that could handle the complexities of a social network, while also providing us with an extensive toolkit to build out all the features we envisioned. Our frontend is built using [React](https://reactjs.org), a powerful and well-supported web framework that gives us the flexibility to build with ease. We utilized supporting frontend technologies like Bootstrap, HTML, and CSS to help create an attractive UI, the key aspect of any social media. For the backend, we used [Django](https://www.djangoproject.com) and [Django Rest Framework](https://www.django-rest-framework.org) to build a secure API that our frontend can easily interact with. For our recommendation algorithm, we used scikit-learn and numpy to power our machine learning needs. Finally, we used PostgreSQL for our DBMS and Heroku for deployment.

## Challenges we ran into

As with most social media platforms, users are key. Given the *very short* nature of hackathons, it obviously isn't feasible to attract a large number of users for development purposes. However, we needed a way to have users available for testing. Since ensemble is based on Spotify accounts and the Spotify API, this proved to be non-trivial. We took advantage of the Spotify API's recommendations endpoint to generate pseudo-data that resembles what a real person would have as their top tracks. With a fake name generator, we created as many fake profiles as we needed to flesh out our recommendation algorithm.

## Accomplishments that we're proud of

Our application is fully ready to use—it has all of the necessary authentication, authorization, and persistent storage. While we'd love to add even more features, we focused on implementing the core ones in their current state (if you use Spotify, feel free to log in and try it out!). You can find the live version [here](https://ensemble-dev.herokuapp.com). Despite all of the hassle of the deployment process, it was very fulfilling to see what we created, live and ready to be used by anyone in the world.

We're also proud of what we've accomplished in general! It's been a challenging yet immensely fulfilling day-and-a-half of ideation, design, and coding. Looking back at what we were able to create during this short time span, we're proud to have something to show for all the effort we've put into it.

## What we learned

We both learned a lot from working on this project. It's been a fast-paced weekend of continuously pushing new changes and features, and in doing so, we sharpened our skills in both React and Django. Additionally, utilizing the Spotify API was something neither of us had done before, and we learned a lot about OAuth 2.0 and web authentication in general.

## What's next for ensemble

Working on this project was a lot of fun, and we'd both love to keep it going in the future. There are a ton of features that we thought out but didn't have the time to implement in this time span. For example, we'd love to implement a direct messaging system, so you can directly contact and discuss your favorite songs/artists with the people you follow. The GitHub repository readme also contains complete and detailed instructions on how to set up your development environment to run the code, if anyone is interested in trying it out. Thanks for reading!
## Intro and Idea

For our team of First Year UofT Engineering Science students, this was our first Makeathon and first project as a team. We have varying ranges of experience with software and hardware within our team and decided to approach this competition as both a challenge, and a learning experience.

After a couple hours of brainstorming based on our collective interests, our team arrived on an idea we were all excited about: An interactive orchestra experience to allow players to more easily play together. Jazz ensembles are easily able to improvise together, because they usually play in the same keys. Classical musicians on the other hand, are often not able to predict key changes.

Our design provides a platform for conductors to change the orchestra in real-time according to their vision. By playing chords on a Midi keyboard, they can “play the orchestra” by transposing and transmitting the chords to the members of the orchestra through the use of wireless connectivity to individual displays powered by Raspberry Pi’s.

## Planning and Summary

As briefly identified in our introduction, our primary stakeholders for this project are:

Ourselves (a team of first year engineering students attempting their first Makeathon)
Orchestra performers and conductors
MakeUofT Organizers, Sponsors, and Judges

Based on this, we were able to develop some rough objectives to keep us on track for 24 hours:

To create a unique but achievable product
To improve our software and hardware integration skills
To incorporate sponsor innovations and technologies

To ensure we had something to show after 24 hours, we decided to aim for a minimum viable project (MVP) before adding any bells and whistles. We were able to reach our goals of Midi communication to the serving computer and having note name communication to the player displays. After we reached our MVP, we expanded on our design to have visualization of the notes on the staff, differing transposition options, and automated chord analysis. Finally, we implemented chord suggestion and prediction using Azure Machine Learning.

## Features:

Real time note communication between Midi and player displays
Visual display of notes on staff
Automated Chord Analysis
Chord suggestion and predictions using Azure Machine Learning
Multiple differently transposed sections available to accommodate a variety of instruments simultaneously

## Applications:

Large group improvisation and composition
Teaching and training
Creating new pieces using Azure Machine Learning

## The Process

## Raspberry Pi Setup and Enclosure:

To act as the receiving devices, we use four Raspberry Pi’s in our project. Each Pi is set up with Raspbian Stretch version 4.14.

A 7” touch display screen is attached to one of our Pi’s, and monitors to the remaining three. Initially, we were going to use small LCD graphic displays, but ruled these out due to size. A 7” touch display was offered to us to borrow, but to keep our cost down, we opted to use monitors for the remaining Pi’s. Ideally all the Pi’s would have 7” touch display screens, but we decided for a MVP prototype, one was sufficient.

Beyond our MVP prototype we added a push-button that allows the player to cycle through the available sections (more on sections to follow) on the Raspberry Pi. A simple python script was created to map the GPIO pin connected to the button to a keyboard stroke (the F5 refresh button for web pages). We left room in the box for more features to be added.

We initially hoped to modify the open source PIvena Raspberry Pi enclosure, but after beginning our modifications, we realized laser cutting was not being offered as a service at MakeUofT. Based on limited fabrication lab hours available to us, we opted to design our enclosure out of foamcore.

The display is set at an angle in order to allow notes to be read easily while playing an instrument. Hardware is located in a box behind the display, making it discrete but easily accessible.

## Network/IoT Setup:

The Network setup remained simple throughout the project. The basic concept was to have one computer that acted as a master, then any number of displays of all shapes and sizes that could receive instructions from the master and join in on the joys of music. To accomplish this, we used a Node.js framework and wrote predominantly JavaScript to manage the interactivity of different clients. As a result of our objectives, the final product that we have produced is capable of being run on any platform with a web browser, making it highly accessible and scalable. Furthermore, as the result works off of a wireless network, it is capable of accepting a high volume of hosts without added latency, as well as being very easy to connect to. The network setup has been geared towards the IoT model by connecting devices in a collaborative way to enable people to help and support each other in harmony.

Technically speaking, the socket works using Socket.io, integrating native HTTPS for requests to the Azure Cloud for learned suggestions of chords based on machine intelligence. The socketing breaks the network into a series of sections, which may all operate and play in different keys, requiring transposition for accessible harmony. The number of sections that can be created is theoretically undefined, though for our basic demonstration we make use of four different sections operating in different keys.They are all shown the classic chord data and have available to them the key in which the orchestra is playing giving the player further liberty for experimentation within the piece.

## Machine Learning:

The Azure Machine Learning framework was the center of a feature for the master controller of the project. Provided with historical data of chords played, it would recommend a good follow-up chord to harmonize. Our machine learning algorithm was fed by approximately 3 million data points from pop songs. Though in retrospect we should have trained it with music that finds more standard and less repetitive chords, the concept still worked well enough, though at our hand there was slight underfitting. The structure that worked well for our means was the feeding of 3 points of historical data predicting a fourth point of data to a fair degree of reason. This is a useful feature of our design for those who would use this concept to collaborate with others or create something new, as it would support them further in a desire for a good, harmonic sound.