## TL; DR

* Music piracy costs the U.S. economy [$12.5 billion annually](https://www.riaa.com/wp-content/uploads/2015/09/20120515_SoundRecordingPiracy.pdf).
* Independent artists are the [fastest growing segment in the music industry](https://www.forbes.com/sites/melissamdaniels/2019/07/10/for-independent-musicians-goingyour-own-way-is-finally-starting-to-pay-off/), yet lack the funds and reach to enforce the Digital Millennium Copyright Act (DMCA).
* We let artists **OWN** their work (stored on InterPlanetary File System) by tracking it on our own Sonoverse Ethereum L2 chain (powered by Caldera).
* Artists receive **Authenticity Certificates** of their work in the form of Non-Fungible Tokens (NFTs), powered by Crossmint’s Minting API.
* We protect against parodies and remixes with our **custom dual-head LSTM neural network model** trained from scratch which helps us differentiate these fraudulent works from originals.
* We proactively query YouTube through their API to constantly find infringing work.
* We’ve integrated with **DMCA Services**, LLC. to automate DMCA claim submissions.

Interested? Keep reading!

## Inspiration

Music piracy, including illegal downloads and streaming, costs the U.S. economy $12.5 billion annually.
Independent artists are the fastest growing segment in the music industry, yet lack the funds to enforce DMCA.

We asked “Why hasn’t this been solved?” and took our hand at it. Enter Sonoverse, a platform to ensure small musicians can own their own work by automating DMCA detection using deep learning and on-chain technologies.

## The Issue

* Is it even possible to automate DMCA reports?
* How can a complex piece of data like an audio file be meaningfully compared?
* How do we really know someone OWNS an audio file?
* and more...

These are questions we had too, but by making custom DL models and chain algorithms, we have taken our hand at answering them.

## What we’ve made

We let artists upload their original music to our platform where we store it on decentralized storage (IPFS) and our blockchain to **track ownership**. We also issue Authenticity Certificates to the original artists in the form of Non-Fungible Tokens.

We compare uploaded music with all music on our blockchain to **detect** if it is a parody, remix, or other fraudulent copy of another original song, using audio processing and an LSTM deep learning model built and trained from scratch.

We proactively query YouTube through their API for “similar” music (based on our **lyric hashes**, **frequency analysis**, and more) to constantly find infringing work. For detected infringing work, we’ve integrated with DMCA Services, LLC. to **automate DMCA claim submissions**.

## How we built it

All together, we used…

* NextJS
* Postgres
* AWS SES
* AWS S3
* IPFS
* Caldera
* Crossmint
* AssemblyAI
* Cohere
* YouTube API
* DMCA Services

It’s a **lot**, but we were able to split up the work between our team. Gashon built most of the backend routes, an email magic link Auth platform, DB support, and AWS integrations.

At the same time, Varun spent his hours collecting hours of audio clips, training and improving the deep LSTM model, and writing several sound differentiation/identification algorithms. Here’s Varun’s **explanation** of his algorithms: “To detect if a song is a remix, we first used a pre-trained speech to text model to extract lyrics from mp3 files and then analyzed the mel-frequency cepstral coefficients, tempo, melody, and semantics of the lyrics to determine if any songs are very similar. Checking whether a song is a parody is much more nuanced, and we trained a dual-head LSTM neural network model in PyTorch to take in vectorized embeddings of lyrics and output the probability of one of the songs being a parody of the other.”

While Varun was doing that, Ameya built out the blockchain services with Caldera and Crossmint, and integrated DMCA Services. Ameya ran a Ethereum L2 chain specific for this project (check it out [here](https://treehacks-2024.explorer.caldera.xyz)) using Caldera. He built out significant infrastructure to upload audio files to IPFS (decentralized storage) and interact with the Caldera chain. He also created the Authenticity Certificate using Crossmint that’s delivered directly to each Sonoverse user’s account.

Ameya and Gashon came together at the end to create the Sonoverse frontend, while Varun pivoted to create our YouTube API jobs that query through recently uploaded videos to find infringing content.

## Challenges we overcame

We couldn’t find existing models to detect parodies and had to train a custom model from scratch on training data we had to find ourselves. Of course, this was quite challenging, but with audio files each being unique, we had to create a dataset of hours of audio clips.

And, like always, integration was difficult. The power of a team was a huge plus, but also a challenge. Ameya’s blockchain infrastructure had Solidity compilation challenges when porting into Gashon’s platform (which took some precious hours to sort out). Varun’s ML algorithms ran on a Python backend which had to be hosted alongside our NextJS platform. You can imagine what else we had to change and fix and update, so I won’t bore you.

Another major challenge was something we brought on ourselves, honestly. We set our aim high so we had to use several different frameworks, services, and technologies to add all the features we wanted. This included several hours of us learning new technologies and services, and figuring out how to implement them in our project.

## Accomplishments that we're proud of

Blockchain has a lot of cool and real-world applications, but we’re excited to have settled on Sonoverse. We identified a simple (yet technically complex) way to solve a problem that affects many small artists.
We also made a sleek web platform, in just a short amount of time, with scalable endpoints and backend services.

We also designed and trained a deep learning LSTM model to identify original audios vs fraudulent ones (remixes, speed ups, parodies, etc) that achieved **93% accuracy**.

## What we learned

#### About DMCA

We learned how existing DMCA processes are implemented and the large capital costs associated with them. We became **experts** on digital copyrights and media work!

#### Blockchain

We learned how to combine centralized and decentralized infrastructure solutions to create a cohesive **end-to-end** project.

## What's next for Sonoverse

We're looking forward to incorporating on-chain **royalties** for small artists by detecting when users consume their music and removing the need for formal contracts with big companies to earn revenue.
We’re excited to also add support for more public APIs in addition to YouTube API!
# Team

Honeycrisp

# Inspiration

Every year there are dozens of heatstroke accidents that occur, a number of which are defined as vehicular heatstroke accidents. Our aim was to build a device for vehicles to best prevent these scenarios, whether there may be young children or pets left in said vehicles.

# What it does

Detector that detects temperature/environment conditions within a car and presence of any living being, so as to alert the owner when the environment reaches dangerous conditions for any living beings inside the vehicle (babies, pets, ...)

# How the Detector Works

The detector makes use of several sensors to determine whether the environmental conditions within a vehicle have reached dangerous levels, and whether there is presence of a living being within the vehicle. In the case where both are true it is to send a text message to the owner of the car warning them about the situation within the vehicle.

# How we built it

A team of 3 people made use of the Particle Electron board and several sensors (Gas sensors, Thermal sensors, Infrared Motion sensor as well as Audio Sensor) to create the project.

# Challenges we faced

There were challenges faced when dealing with the Particle Electron board, in that the sensors being used were made for an Arduino. This required specific libraries, which eventually caused the Particle Electron board to malfunction.

# Accomplishments

The team has no past experience working with a Particle Electron board, so for the work that was accomplished within the 24 hour span, we consider it a success.

# What we learned

We learned a lot about the Particle Electron board as well as the sensors that were utilized for this project

# Future

Future developments to improve our device further would include:

1. Considering sensors with more precision to ensure that the conditions and parameters being monitored are as
precise as required.
2. Implement multiple emergency measures, in the case where reaching the owner becomes difficult or the
conditions within the vehicle have reached alarming levels:
a. Turning on the A/C of the vehicle
b. Cracking the window slightly open for better circulation.
c. Have the vehicle make noise (either via the alarm system or car horn) to gain the attention of any
 passerby or individuals within reasonable distances to call for aid.
d. Function that reports the incidence to 911, along with the location of the vehicle.
## Inspiration

We wanted to create a webapp that will help people learn American Sign Language.

## What it does

SignLingo starts by giving the user a phrase to sign. Using the user's webcam, gets the input and decides if the user signed the phrase correctly. If the user signed it correctly, goes on to the next phrase. If the user signed the phrase incorrectly, displays the correct signing video of the word.

## How we built it

We started by downloading and preprocessing a word to ASL video dataset.
We used OpenCV to process the frames from the images and compare the user's input video's frames to the actual signing of the word. We used mediapipe to detect the hand movements and tkinter to build the front-end.

## Challenges we ran into

We definitely had a lot of challenges, from downloading compatible packages, incorporating models, and creating a working front-end to display our model.

## Accomplishments that we're proud of

We are so proud that we actually managed to build and submit something. We couldn't build what we had in mind when we started, but we have a working demo which can serve as the first step towards the goal of this project. We had times where we thought we weren't going to be able to submit anything at all, but we pushed through and now are proud that we didn't give up and have a working template.

## What we learned

While working on our project, we learned a lot of things, ranging from ASL grammar to how to incorporate different models to fit our needs.

## What's next for SignLingo

Right now, SignLingo is far away from what we imagined, so the next step would definitely be to take it to the level we first imagined. This will include making our model be able to detect more phrases to a greater accuracy, and improving the design.