GCC-PHAT (Generalized cross correlation) always peak at delay=0 on real audio signal
I think that's because of the DC offset of the signal. A time domain or frequency domain plot provided by you could have been helpful here for the confirmation of your doubt. Following are suggestions

1.try to filter the signals before FFT depending upon the signal band you are interested in

2.Subtract the signal by the mean of the entire signal (which is a reasonable trick to find DC offset) or by the DC offset directly if you know that.

The signal of your's must have less swing amplitude in comparison to DC offset making the '0' point correlation of greater amplitude than the actual delay.