Problem with Generalized Cross Correlation (PHAT filter)
Using the gcc-phat method, you can use the matlab function "gccphat(s2,s1,fs)" (R2016a) with s1 as the reference signal, s2 the other one and fs the sampling frequency. 

Basically it will do a similar correlation calculation :

  Ncorr = 2*N-1;  % N size of the signal
  NFFT = 2^nextpow2(Ncorr);
  R12 = bsxfun(@times,fft(s2,NFFT),conj(fft(s1,NFFT)));
  r12_temp = fftshift(ifft(exp(1i*angle(R12))),1);
  r12 = r12_temp(NFFT/2+1-(N-1)/2:NFFT/2+1+(N-1)/2,:);


The only difference with your code is that the matlab function use the nearest power of two sequence length for the fft (as Peter.K suggest with "Nfft = length(x1) + length(x2) - 1") and doing an exponentiel stuff (and this part make the difference with a regular correlation, what you were doing)

But to return the time delay, the function is doing an other trick :

  lags = (-(Ncorr-1)/2:(Ncorr-1)/2).';
  lags = lags/fs;
  [~,idx] = max(abs(r12));
  tau = ts/(2*fs)+lags(idx); % ts as the total sample of the signal


I'm doing similar experimentation and this method is working great for long and short non-moving signal. the reverberation of the room may cause some issue, but if you can see an obvious delay when displaying the signal it should work fine.

Right now, i'm having difficulty to return a good time delay with a moving source (the function retourn 0 or +/- the length of the signal) And i'm still working on it... So if anybody got a lead, it will be much apreciate ;-)

Hope this will help you.

Peace

PS : Also, when doing an fft/ifft you really should use fftshift/ifftshift because the matlab fft/ifft is mixing some stuff during the process.