Sliding DFT: I think my results are accurate, but I can't buy a swap-swap IDFT
I haven't tried to run your code (is that C#?) or looked at it in great detail, but SlidingDFT() looks more like a Goertzel algorithm implementation to me. It should be a red flag to you that you're able to perform the calculations in place. If you were truly doing a sliding DFT, then you would have a data rate expansion by a factor of $N$, the length of the transform; for each input time-domain sample, you generate a full DFT's worth of outputs.

The Goertzel algorithm is an efficient, recursive structure for implementing a single DFT bin at an arbitrary center frequency, which can be useful for applications like DTMF decoding. However, you can't "invert" it in the sense that you're trying to; it is more of a filtering operation, not a reversible transform. 

It doesn't seem to make sense to me to even try to invert a sliding DFT in the way you've shown; the outputs of a sliding DFT don't "slide" sample-by-sample the same way as the time-domain inputs do. I'd read up some more on how the DFT and its sliding version works.