Confusion with data after cross correlation
I had done something like this and ran into similar problems a while ago, using Matlab. The audio to the microphones was speech or whistling in my case. I was estimating the delay between two mics to find the direction sound was arriving from.

The audio data from the mics probably has oscillations of some kind (like a sum of sinusoids plus some noise), at least at certain locations. The cross-correlation of one such waveform with a delayed version of it will be a signal with oscillations. This would look like this example, but possibly decaying slowly. I've seen something similar in this question.

The cross-correlation peak should indicate the delay, but there are multiple peaks so you may be detecting the wrong one. Since your input signal is not a pure sine-wave, the largest peak should be the delay but it would be hard to guarantee that in a hardware system. This may be why you're getting inconsistent results.

Since you know the distance between the two microphones (and the system's sampling rate), you can estimate the maximum expected delay (in a number of samples), as the time it would take for sound to travel the full distance between the mics (0.6 meters). Then you should search for the cross-correlation peak only in the first such number samples, rather than in the full waveform.

Low-pass filtering may also be helpful if you're just interested in the delay. The higher frequency components can create peaks in the cross-correlation sooner (oscillations with a smaller period). In my case, I had found that a low-pass cut-off of about 1.0-1.5 kHz was adequate since my signal (speech) was wide-band and contained enough information at low frequencies for the delay estimation.