Spectrogram of an executable binary
This serves as potential starting points only, on how to transpose classical signal processing tools to specific kinds of binary data. 

First, remember that standard spectrograms, fed as inputs to classifiers, can be seen as means to extract attributes from non-stationary data. There is apparently little need for inversion, so we are interested in analysis only (no synthesis). 

Here, we have sequences of 8-bit words $w_0w_1w_2\ldots w_{N-1}$. When dealing with binary patterns, the frequency domain might not be the most appropriate. In that case, as with communication systems, the related notion of sequency could be useful. It relates to blocky oscillating patterns of $\pm 1$ (or $0/1$):



The number of zero-crossing increases in a manner similar to augmenting frequencies in Fourier transforms. I would thus prefer to use those vectors in sliding windows to analyze your binaries. You can find literature in such functions as Walsh, Paley or Hadamard (or WaLeyMard) sequences, depending on their ordering. The proper choice of the transform length is not evident.  This seems to exist already, under the name: "Walsh spectrogram": On Generating Walsh Spectrograms, 1976.

I would think it might depend a bit (pun intended) on the compiling or storage system. There is a possibility to look at dyadic size factors with Haar wavelets or Haar wavelet packets. 

Note that there exist transformations for data taking values in finite fields, from The Fast Fourier Transform in a Finite Field, 1971, J. M. Pollard.

Other references:


Harmut, H. F., 1970: Transmission of information by orthogonal functions
Agaian, S. S., 1984: Hadamard Matrices and Their Applications