Testing for changes in auto-covariance
I'd recommend looking into the relation of correlation and covariance; the one is just the "bias-corrected" version of the other.

Then, use the Wiener-Khinchin¹ theorem: If, and only if, the signal is weak-sense stationary, the Fourier transform of the autocorrelation of that signal is the same as Expectation value of the magnitude-squared Fourier transform of the signal itself.

So, I think, you're on the right track: Just use (potentially even overlapping) Short-term Fourier Transforms to get an estimate for the Fourier transform of the autocorrelation; if you know the mean is fixed, that should be hold the same amount of information as auto-covariance. Compare those over time.

In fact, with what is often known as a spectrogram in plotting libraries, you'd get something that visually represents that pretty well. 


¹ I say this every time: There's more spellings to Tschinshin's name than there's letters in his originally cyrillic name.