In music information retrieval, one usually converts an audio signal into some kind "sequence of frequency-vectors", such as STFT or Mel-spectrogram.

I'm wondering if it is a good idea to use the transformer architecture in a self-supervised manner -- such as auto-regressive models, or BERT in NLP -- to obtain a "smarter" representation of the music than the spectrogram itself. Such smart pretrained representation could be used for further downstream tasks.

From my quick google search, I found several papers which do something similar, but -- to my surprise -- all use some kind of symbolic/discrete music representation such as scores. (For instance [here](https://arxiv.org/pdf/1809.04281.pdf) or [here](https://arxiv.org/pdf/1912.05537.pdf)).

My question is this:

> 
> Is it realistic to train such an unsupervised model directly on the
> Mel spectrogram?
> 
> 
> 

The loss function would not be "log softmax of next word probability", but some kind of l2-distance between "predicted vector of spectra" and "observed vector of spectra", in the next time step.

Did someone try it?