Recovering phase from spectrogram
It should be clear from the definition of the STFT that the result is invariant under global phase changes. With the restriction to real input signals that still provides for a sign ambiguity, or a phase difference of $\pi$. That means, even if your reconstruction is perfect, the result may differ by that phase from the original.

Another reason for the failure to converge may be the existence of nearly disconnected components in your signal. Say you have two sound components that are temporally separated so that there is no STFT frame that overlaps with both. Each component may be reconstructed with a different sign, giving you four different reconstructions with identical STFT. The same works for components that are separated well enough in frequency direction. Each component doubles the ambiguity of the reconstructed signal and will increase the probability of a deviation from the original signal.

The final reason I can see lies in the nature of the iterative algorithm. Fixed point iterations, of which kind this algorithm is, can fail to converge to a proper fixed point, keep oscillating or even diverge. Unless the maps involved are well behaved in a certain sense, predicting convergence is hard. It seems that your iteration doesn't settle, and it's unclear if adding iterations will help. So this may apply here, together with the other two problems.