Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram? In music information retrieval, one usually converts an audio signal into some kind &quot;sequence of frequency-vectors&quot;, such as STFT or Mel-spectrogram.
I'm wondering if it is a good idea to use the transformer architecture in a self-supervised manner -- such as auto-regressive models, or BERT in NLP -- to obtain a &quot;smarter&quot; representation of the music than the spectrogram itself. Such smart pretrained representation could be used for further downstream tasks.
From my quick google search, I found several papers which do something similar, but -- to my surprise -- all use some kind of symbolic/discrete music representation such as scores. (For instance here or here).
My question is this:

Is it realistic to train such an unsupervised model directly on the
Mel spectrogram?

The loss function would not be &quot;log softmax of next word probability&quot;, but some kind of l2-distance between &quot;predicted vector of spectra&quot; and &quot;observed vector of spectra&quot;, in the next time step.
Did someone try it?
The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that MAESTRO (the dataset behind the two papers you linked) contains performances, not scores, so they include timing and accents of real performers--but are still sequences of discrete events, not audio.
There's been some work on learning discrete representations directly from audio, such as with vector-quantized variational autoencoders (VQ-VAEs). Typically an autoregressive model is trained on top of the learned representation; Jukebox used a transformer for that. By the way, I'd highly recommend reading through the &quot;related work&quot; section of the Jukebox paper for an overview of work on the audio/speech synthesis task.
wav2vec is probably closest to what you're describing. They train a transformer on raw audio, self-supervised, in order to learn good representations of human speech for the speech-to-text task.
As far as training directly on spectograms, there's MelNet, a somewhat exotic RNN trained for a variety of audio synthesis tasks, including music.
Hope this helps!