Google recently demonstrated how MusicLM can compose high-fidelity audio from a text prompt with generative AI technology. Let's make some predictions about how AI tech will affect the music industry in the future. 

#ai #tech #TheCodeReport

💬 Chat with Me on Discord

https://discord.gg/fireship

🔗 Resources

MusicLM https://google-research.github.io/seanet/musiclm/examples
MusicCaps Dataset https://www.kaggle.com/datasets/googleai/musiccaps
Riffusion https://www.riffusion.com


🔥 Get More Content - Upgrade to PRO

Upgrade at https://fireship.io/pro
Use code YT25 for 25% off PRO access 

🎨 My Editor Settings

- Atom One Dark 
- vscode-icons
- Fira Code Font

🔖 Topics Covered

- Can AI generate music?
- Voice cloning
- ChatGPT for writing songs
- How Artificial Music works
- Google AudioLM
- Google MusicLM
- Machine learning with audio
2024-02-04T05:13:19.857Z
nova-2
2-general-nova
2024-01-09.29447
30089e05-99d1-4376-b32e-c263170674af
3eb3ce75-b99e-4122-8987-dea4d6818695
8c4456b1ba3351b9108d0a84825c21428cf91bb07ab0a004c0fe7517193cd8f1
67875a7f-c9c4-48a0-aa55-5bdb8a91c34a
deprecated
It is January 31, 2023 and you're watching the code report.
Beethoven once said music can change the world, but I really hope this dude was wrong because AI is getting pretty good at making its own music.
This relaxing jazz track you're hearing in the background right now isn't Kenny g, but rather a 5 minute AI generated jam session created with a simple text prompt.
Music is the strongest form of magic.
It transcends all language barriers, yet humans are the only creatures that create and enjoy it, with the exception of parrots, of course.
Google just released a paper called MusicLM, and it describes its research into generating music from a text prompt.
It's basically a chat GPT or DALL E for music.
Of all digital art forms, music is the most difficult to reproduce because it takes place of a specific temporal space consisting of thousands of tokens and needs to invoke human emotions.
That's why every morning I listen to eye of the tiger to get fired up before writing any JavaScript code, but listening to the same song over and over again gets boring.
Let's find out if AI can generate new music for us on the fly.
This model was trained on a relatively small dataset of 5,000 music samples paired with a text description, and you can access this data on Kaggle.
Before we talk about it though, you should know that there's another groundbreaking text to music project out there called Refusion.
It works like a text to image generator, except that it creates spectrograms, which are then convert it to audio and actually sound like coherent music.
That's pretty cool, but MusicLM takes a completely different approach that doesn't rely on diffusion.
MusicLM is a pure audio model, which relies on 2 different types of tokens.
Semantic tokens capture details like melody and rhythm, while acoustic tokens from Soundstream capture things like the recording conditions of the track.
These tokens are combined, then decoded into a high fidelity audio track.
Here we have a prompt for the soundtrack of an arcade game.
If that were playing in the background of a intro video game.
I'd have no idea it was AI generated.
But now let's try something a little more difficult, a hip hop track with vocals.
I wouldn't say it's a bop, but it's not terrible either.
If you listen closely to the lyrics, they're just nonsense words, but they sound good.
If I were faded at the club, I Easily bust a groove to this shit.
What's really amazing about Music Amelbo is melody conditioning.
Or in other words, you can generate music that respects a text prompt while also providing the actual melody.
Woody.
Like if you record yourself humming Beethoven's Ode to Joy, you can then convert it to any style like an opera singer or guitar solo.
In addition, they tested it on descriptions of paintings, and it does a pretty good job of capturing the mood of the paintings.
Till nowhere near as beautiful as what a talented human musician could create.
World.
But I think AI has the potential to revolutionize music production in the near future.
Imagine a smart keyboard with I like this built in.
You play a little melody, then it comes up with an infinite number of possible arrangements with different sounds, instruments, and styles.
Instead instead of actually composing and producing music, people will just guide the AI in the right direction until the track is bumping.
With continuation and voice cloning, you could bring Michael Jackson and Kurt Cobain back to sing a duet on your track.
You could then use chat gpt to write the perfect lyrics.
And if you're a DJ, you could use vision AI to capture the mood on people's faces in route and generate music on the fly that suits them perfectly.
This has been the Code Report.
Thanks for watching, and I will see you in the next one.

Speaker 0: It is January 31, 2023 and you're watching the code report. Beethoven once said music can change the world, but I really hope this dude was wrong because AI is getting pretty good at making its own music. This relaxing jazz track you're hearing in the background right now isn't Kenny g, but rather a 5 minute AI generated jam session created with a simple text prompt. Music is the strongest form of magic. It transcends all language barriers, yet humans are the only creatures that create and enjoy it, with the exception of parrots, of course.

Google just released a paper called MusicLM, and it describes its research into generating music from a text prompt. It's basically a chat GPT or DALL E for music. Of all digital art forms, music is the most difficult to reproduce because it takes place of a specific temporal space consisting of thousands of tokens and needs to invoke human emotions. That's why every morning I listen to eye of the tiger to get fired up before writing any JavaScript code, but listening to the same song over and over again gets boring. Let's find out if AI can generate new music for us on the fly.

This model was trained on a relatively small dataset of 5,000 music samples paired with a text description, and you can access this data on Kaggle. Before we talk about it though, you should know that there's another groundbreaking text to music project out there called Refusion. It works like a text to image generator, except that it creates spectrograms, which are then convert it to audio and actually sound like coherent music. That's pretty cool, but MusicLM takes a completely different approach that doesn't rely on diffusion. MusicLM is a pure audio model, which relies on 2 different types of tokens.

Semantic tokens capture details like melody and rhythm, while acoustic tokens from Soundstream capture things like the recording conditions of the track. These tokens are combined, then decoded into a high fidelity audio track. Here we have a prompt for the soundtrack of an arcade game. If that were playing in the background of a intro video game. I'd have no idea it was AI generated.

But now let's try something a little more difficult, a hip hop track with vocals. I wouldn't say it's a bop, but it's not terrible either. If you listen closely to the lyrics, they're just nonsense words, but they sound good. If I were faded at the club, I Easily bust a groove to this shit. What's really amazing about Music Amelbo is melody conditioning.

Or in other words, you can generate music that respects a text prompt while also providing the actual melody. Woody. Like if you record yourself humming Beethoven's Ode to Joy, you can then convert it to any style like an opera singer or guitar solo. In addition, they tested it on descriptions of paintings, and it does a pretty good job of capturing the mood of the paintings. Till nowhere near as beautiful as what a talented human musician could create.

World. But I think AI has the potential to revolutionize music production in the near future. Imagine a smart keyboard with I like this built in. You play a little melody, then it comes up with an infinite number of possible arrangements with different sounds, instruments, and styles. Instead instead of actually composing and producing music, people will just guide the AI in the right direction until the track is bumping.

With continuation and voice cloning, you could bring Michael Jackson and Kurt Cobain back to sing a duet on your track. You could then use chat gpt to write the perfect lyrics. And if you're a DJ, you could use vision AI to capture the mood on people's faces in route and generate music on the fly that suits them perfectly. This has been the Code Report. Thanks for watching, and I will see you in the next one.
It is January 31, 2023 and you're watching the code report. Beethoven once said music can change the world, but I really hope this dude was wrong because AI is getting pretty good at making its own music. This relaxing jazz track you're hearing in the background right now isn't Kenny g, but rather a 5 minute AI generated jam session created with a simple text prompt. Music is the strongest form of magic. It transcends all language barriers, yet humans are the only creatures that create and enjoy it, with the exception of parrots, of course. Google just released a paper called MusicLM, and it describes its research into generating music from a text prompt. It's basically a chat GPT or DALL E for music. Of all digital art forms, music is the most difficult to reproduce because it takes place of a specific temporal space consisting of thousands of tokens and needs to invoke human emotions.
music
sounds
audio
podcasts
That's why every morning I listen to eye of the tiger to get fired up before writing any JavaScript code, but listening to the same song over and over again gets boring. Let's find out if AI can generate new music for us on the fly. This model was trained on a relatively small dataset of 5,000 music samples paired with a text description, and you can access this data on Kaggle. Before we talk about it though, you should know that there's another groundbreaking text to music project out there called Refusion. It works like a text to image generator, except that it creates spectrograms, which are then convert it to audio and actually sound like coherent music. That's pretty cool, but MusicLM takes a completely different approach that doesn't rely on diffusion.
music
audio
sounds
multimedia
MusicLM is a pure audio model, which relies on 2 different types of tokens. Semantic tokens capture details like melody and rhythm, while acoustic tokens from Soundstream capture things like the recording conditions of the track. These tokens are combined, then decoded into a high fidelity audio track.
Here we have a prompt for the soundtrack of an arcade game. If that were playing in the background of a intro video game. I'd have no idea it was AI generated. But now let's try something a little more difficult, a hip hop track with vocals. I wouldn't say it's a bop, but it's not terrible either. If you listen closely to the lyrics, they're just nonsense words, but they sound good. If I were faded at the club, I Easily bust a groove to this shit. What's really amazing about Music Amelbo is melody conditioning. Or in other words, you can generate music that respects a text prompt while also providing the actual melody. Woody. Like if you record yourself humming Beethoven's Ode to Joy, you can then convert it to any style like an opera singer or guitar solo. In addition, they tested it on descriptions of paintings, and it does a pretty good job of capturing the mood of the paintings. Till nowhere near as beautiful as what a talented human musician could create. World. But I think AI has the potential to revolutionize music production in the near future. Imagine a smart keyboard with I like this built in. You play a little melody, then it comes up with an infinite number of possible arrangements with different sounds, instruments, and styles. Instead instead of actually composing and producing music, people will just guide the AI in the right direction until the track is bumping. With continuation and voice cloning, you could bring Michael Jackson and Kurt Cobain back to sing a duet on your track. You could then use chat gpt to write the perfect lyrics. And if you're a DJ, you could use vision AI to capture the mood on people's faces in route and generate music on the fly that suits them perfectly. This has been the Code Report. Thanks for watching, and I will see you in the next one.
music
singing
video gaming
art
sounds
audio
performing arts
youtube
multimedia
visualization
artificial intelligence
It is January 31, 2023 and you're watching the code report. Beethoven once said music can change the world, but I really hope this dude was wrong because AI is getting pretty good at making its own music. This relaxing jazz track you're hearing in the background right now isn't Kenny g, but rather a 5 minute AI generated jam session created with a simple text prompt. Music is the strongest form of magic. It transcends all language barriers, yet humans are the only creatures that create and enjoy it, with the exception of parrots, of course. Google just released a paper called MusicLM, and it describes its research into generating music from a text prompt. It's basically a chat GPT or DALL E for music. Of all digital art forms, music is the most difficult to reproduce because it takes place of a specific temporal space consisting of thousands of tokens and needs to invoke human emotions. That's why every morning I listen to eye of the tiger to get fired up before writing any JavaScript code, but listening to the same song over and over again gets boring. Let's find out if AI can generate new music for us on the fly. This model was trained on a relatively small dataset of 5,000 music samples paired with a text description, and you can access this data on Kaggle. Before we talk about it though, you should know that there's another groundbreaking text to music project out there called Refusion. It works like a text to image generator, except that it creates spectrograms, which are then convert it to audio and actually sound like coherent music. That's pretty cool, but MusicLM takes a completely different approach that doesn't rely on diffusion. MusicLM is a pure audio model, which relies on 2 different types of tokens. Semantic tokens capture details like melody and rhythm, while acoustic tokens from Soundstream capture things like the recording conditions of the track. These tokens are combined, then decoded into a high fidelity audio track. Here we have a prompt for the soundtrack of an arcade game. If that were playing in the background of a intro video game. I'd have no idea it was AI generated. But now let's try something a little more difficult, a hip hop track with vocals. I wouldn't say it's a bop, but it's not terrible either. If you listen closely to the lyrics, they're just nonsense words, but they sound good. If I were faded at the club, I Easily bust a groove to this shit. What's really amazing about Music Amelbo is melody conditioning. Or in other words, you can generate music that respects a text prompt while also providing the actual melody. Woody. Like if you record yourself humming Beethoven's Ode to Joy, you can then convert it to any style like an opera singer or guitar solo. In addition, they tested it on descriptions of paintings, and it does a pretty good job of capturing the mood of the paintings. Till nowhere near as beautiful as what a talented human musician could create. World. But I think AI has the potential to revolutionize music production in the near future. Imagine a smart keyboard with I like this built in. You play a little melody, then it comes up with an infinite number of possible arrangements with different sounds, instruments, and styles. Instead instead of actually composing and producing music, people will just guide the AI in the right direction until the track is bumping. With continuation and voice cloning, you could bring Michael Jackson and Kurt Cobain back to sing a duet on your track. You could then use chat gpt to write the perfect lyrics. And if you're a DJ, you could use vision AI to capture the mood on people's faces in route and generate music on the fly that suits them perfectly. This has been the Code Report. Thanks for watching, and I will see you in the next one.
It
it
is
is
January
january
31,
31
2023
2023
and
and
you're
you're
watching
watching
the
the
code
code
report.
report
Beethoven
beethoven
once
once
said
said
music
music
can
can
change
change
the
the
world,
world
but
but
I
i
really
really
hope
hope
this
this
dude
dude
was
was
wrong
wrong
because
because
AI
ai
is
is
getting
getting
pretty
pretty
good
good
at
at
making
making
its
its
own
own
music.
music
This
this
relaxing
relaxing
jazz
jazz
track
track
you're
you're
hearing
hearing
in
in
the
the
background
background
right
right
now
now
isn't
isn't
Kenny
kenny
g,
g
but
but
rather
rather
a
a
5
5
minute
minute
AI
ai
generated
generated
jam
jam
session
session
created
created
with
with
a
a
simple
simple
text
text
prompt.
prompt
Music
music
is
is
the
the
strongest
strongest
form
form
of
of
magic.
magic
It
it
transcends
transcends
all
all
language
language
barriers,
barriers
yet
yet
humans
humans
are
are
the
the
only
only
creatures
creatures
that
that
create
create
and
and
enjoy
enjoy
it,
it
with
with
the
the
exception
exception
of
of
parrots,
parrots
of
of
course.
course
Google
google
just
just
released
released
a
a
paper
paper
called
called
MusicLM,
musiclm
and
and
it
it
describes
describes
its
its
research
research
into
into
generating
generating
music
music
from
from
a
a
text
text
prompt.
prompt
It's
it's
basically
basically
a
a
chat
chat
GPT
gpt
or
or
DALL
dall
E
e
for
for
music.
music
Of
of
all
all
digital
digital
art
art
forms,
forms
music
music
is
is
the
the
most
most
difficult
difficult
to
to
reproduce
reproduce
because
because
it
it
takes
takes
place
place
of
of
a
a
specific
specific
temporal
temporal
space
space
consisting
consisting
of
of
thousands
thousands
of
of
tokens
tokens
and
and
needs
needs
to
to
invoke
invoke
human
human
emotions.
emotions
That's
that's
why
why
every
every
morning
morning
I
i
listen
listen
to
to
eye
eye
of
of
the
the
tiger
tiger
to
to
get
get
fired
fired
up
up
before
before
writing
writing
any
any
JavaScript
javascript
code,
code
but
but
listening
listening
to
to
the
the
same
same
song
song
over
over
and
and
over
over
again
again
gets
gets
boring.
boring
Let's
let's
find
find
out
out
if
if
AI
ai
can
can
generate
generate
new
new
music
music
for
for
us
us
on
on
the
the
fly.
fly
This
this
model
model
was
was
trained
trained
on
on
a
a
relatively
relatively
small
small
dataset
dataset
of
of
5,000
5,000
music
music
samples
samples
paired
paired
with
with
a
a
text
text
description,
description
and
and
you
you
can
can
access
access
this
this
data
data
on
on
Kaggle.
kaggle
Before
before
we
we
talk
talk
about
about
it
it
though,
though
you
you
should
should
know
know
that
that
there's
there's
another
another
groundbreaking
groundbreaking
text
text
to
to
music
music
project
project
out
out
there
there
called
called
Refusion.
refusion
It
it
works
works
like
like
a
a
text
text
to
to
image
image
generator,
generator
except
except
that
that
it
it
creates
creates
spectrograms,
spectrograms
which
which
are
are
then
then
convert
convert
it
it
to
to
audio
audio
and
and
actually
actually
sound
sound
like
like
coherent
coherent
music.
music
That's
that's
pretty
pretty
cool,
cool
but
but
MusicLM
musiclm
takes
takes
a
a
completely
completely
different
different
approach
approach
that
that
doesn't
doesn't
rely
rely
on
on
diffusion.
diffusion
MusicLM
musiclm
is
is
a
a
pure
pure
audio
audio
model,
model
which
which
relies
relies
on
on
2
2
different
different
types
types
of
of
tokens.
tokens
Semantic
semantic
tokens
tokens
capture
capture
details
details
like
like
melody
melody
and
and
rhythm,
rhythm
while
while
acoustic
acoustic
tokens
tokens
from
from
Soundstream
soundstream
capture
capture
things
things
like
like
the
the
recording
recording
conditions
conditions
of
of
the
the
track.
track
These
these
tokens
tokens
are
are
combined,
combined
then
then
decoded
decoded
into
into
a
a
high
high
fidelity
fidelity
audio
audio
track.
track
Here
here
we
we
have
have
a
a
prompt
prompt
for
for
the
the
soundtrack
soundtrack
of
of
an
an
arcade
arcade
game.
game
If
if
that
that
were
were
playing
playing
in
in
the
the
background
background
of
of
a
a
intro
intro
video
video
game.
game
I'd
i'd
have
have
no
no
idea
idea
it
it
was
was
AI
ai
generated.
generated
But
but
now
now
let's
let's
try
try
something
something
a
a
little
little
more
more
difficult,
difficult
a
a
hip
hip
hop
hop
track
track
with
with
vocals.
vocals
I
i
wouldn't
wouldn't
say
say
it's
it's
a
a
bop,
bop
but
but
it's
it's
not
not
terrible
terrible
either.
either
If
if
you
you
listen
listen
closely
closely
to
to
the
the
lyrics,
lyrics
they're
they're
just
just
nonsense
nonsense
words,
words
but
but
they
they
sound
sound
good.
good
If
if
I
i
were
were
faded
faded
at
at
the
the
club,
club
I
i
Easily
easily
bust
bust
a
a
groove
groove
to
to
this
this
shit.
shit
What's
what's
really
really
amazing
amazing
about
about
Music
music
Amelbo
amelbo
is
is
melody
melody
conditioning.
conditioning
Or
or
in
in
other
other
words,
words
you
you
can
can
generate
generate
music
music
that
that
respects
respects
a
a
text
text
prompt
prompt
while
while
also
also
providing
providing
the
the
actual
actual
melody.
melody
Woody.
woody
Like
like
if
if
you
you
record
record
yourself
yourself
humming
humming
Beethoven's
beethoven's
Ode
ode
to
to
Joy,
joy
you
you
can
can
then
then
convert
convert
it
it
to
to
any
any
style
style
like
like
an
an
opera
opera
singer
singer
or
or
guitar
guitar
solo.
solo
In
in
addition,
addition
they
they
tested
tested
it
it
on
on
descriptions
descriptions
of
of
paintings,
paintings
and
and
it
it
does
does
a
a
pretty
pretty
good
good
job
job
of
of
capturing
capturing
the
the
mood
mood
of
of
the
the
paintings.
paintings
Till
till
nowhere
nowhere
near
near
as
as
beautiful
beautiful
as
as
what
what
a
a
talented
talented
human
human
musician
musician
could
could
create.
create
World.
world
But
but
I
i
think
think
AI
ai
has
has
the
the
potential
potential
to
to
revolutionize
revolutionize
music
music
production
production
in
in
the
the
near
near
future.
future
Imagine
imagine
a
a
smart
smart
keyboard
keyboard
with
with
I
i
like
like
this
this
built
built
in.
in
You
you
play
play
a
a
little
little
melody,
melody
then
then
it
it
comes
comes
up
up
with
with
an
an
infinite
infinite
number
number
of
of
possible
possible
arrangements
arrangements
with
with
different
different
sounds,
sounds
instruments,
instruments
and
and
styles.
styles
Instead
instead
instead
instead
of
of
actually
actually
composing
composing
and
and
producing
producing
music,
music
people
people
will
will
just
just
guide
guide
the
the
AI
ai
in
in
the
the
right
right
direction
direction
until
until
the
the
track
track
is
is
bumping.
bumping
With
with
continuation
continuation
and
and
voice
voice
cloning,
cloning
you
you
could
could
bring
bring
Michael
michael
Jackson
jackson
and
and
Kurt
kurt
Cobain
cobain
back
back
to
to
sing
sing
a
a
duet
duet
on
on
your
your
track.
track
You
you
could
could
then
then
use
use
chat
chat
gpt
gpt
to
to
write
write
the
the
perfect
perfect
lyrics.
lyrics
And
and
if
if
you're
you're
a
a
DJ,
dj
you
you
could
could
use
use
vision
vision
AI
ai
to
to
capture
capture
the
the
mood
mood
on
on
people's
people's
faces
faces
in
in
route
route
and
and
generate
generate
music
music
on
on
the
the
fly
fly
that
that
suits
suits
them
them
perfectly.
perfectly
This
this
has
has
been
been
the
the
Code
code
Report.
report
Thanks
thanks
for
for
watching,
watching
and
and
I
i
will
will
see
see
you
you
in
in
the
the
next
next
one.
one
success
The transcript discusses a music project called MusicLM, which uses a text prompt to generate music from a single. The project uses a music model called MusicLM, which uses a pure audio model and uses different tokens to create music. The project has potential to revolutionize music production and is based on AI technology.
The weird future of music
It is January 31, 2023 and you're watching the code report. Beethoven once said music can change the world, but I really hope this dude was wrong because AI is getting pretty good at making its own music. This relaxing jazz track you're hearing in the background right now isn't Kenny g, but rather a 5 minute AI generated jam session created with a simple text prompt. Music is the strongest form of magic. It transcends all language barriers, yet humans are the only creatures that create and enjoy it, with the exception of parrots, of course. Google just released a paper called MusicLM, and it describes its research into generating music from a text prompt. It's basically a chat GPT or DALL E for music. Of all digital art forms, music is the most difficult to reproduce because it takes place of a specific temporal space consisting of thousands of tokens and needs to invoke human emotions. That's why every morning I listen to eye of the tiger to get fired up before writing any JavaScript code, but listening to the same song over and over again gets boring. Let's find out if AI can generate new music for us on the fly. This model was trained on a relatively small dataset of 5,000 music samples paired with a text description, and you can access this data on Kaggle. Before we talk about it though, you should know that there's another groundbreaking text to music project out there called Refusion. It works like a text to image generator, except that it creates spectrograms, which are then convert it to audio and actually sound like coherent music. That's pretty cool, but MusicLM takes a completely different approach that doesn't rely on diffusion. MusicLM is a pure audio model, which relies on 2 different types of tokens. Semantic tokens capture details like melody and rhythm, while acoustic tokens from Soundstream capture things like the recording conditions of the track. These tokens are combined, then decoded into a high fidelity audio track. Here we have a prompt for the soundtrack of an arcade game. If that were playing in the background of a intro video game. I'd have no idea it was AI generated. But now let's try something a little more difficult, a hip hop track with vocals. I wouldn't say it's a bop, but it's not terrible either. If you listen closely to the lyrics, they're just nonsense words, but they sound good. If I were faded at the club, I Easily bust a groove to this shit. What's really amazing about Music Amelbo is melody conditioning. Or in other words, you can generate music that respects a text prompt while also providing the actual melody. Woody. Like if you record yourself humming Beethoven's Ode to Joy, you can then convert it to any style like an opera singer or guitar solo. In addition, they tested it on descriptions of paintings, and it does a pretty good job of capturing the mood of the paintings. Till nowhere near as beautiful as what a talented human musician could create. World. But I think AI has the potential to revolutionize music production in the near future. Imagine a smart keyboard with I like this built in. You play a little melody, then it comes up with an infinite number of possible arrangements with different sounds, instruments, and styles. Instead instead of actually composing and producing music, people will just guide the AI in the right direction until the track is bumping. With continuation and voice cloning, you could bring Michael Jackson and Kurt Cobain back to sing a duet on your track. You could then use chat gpt to write the perfect lyrics. And if you're a DJ, you could use vision AI to capture the mood on people's faces in route and generate music on the fly that suits them perfectly. This has been the Code Report. Thanks for watching, and I will see you in the next one.
The transcript discusses a music project called MusicLM, which uses a text prompt to generate music from a single. The project uses a music model called MusicLM, which uses a pure audio model and uses different tokens to create music. The project has potential to revolutionize music production and is based on AI technology.