DoA estimation using microphone array
The mistake could be in assuming you know which angle $a$ to use when you don't know if the source is in front of or behind the array yet. In other words, you need to solve the same problem for $a$ that you're solving for $b$.

For your approach to work, you need to make sure side $a$ is the closest side to the source. You could find the closest side by finding the pair of mics where the maximum arrival time of the pair is less than the minimum arrival time of the opposite pair. This beam points outward, and the beam on the opposite side points inward. Only one pair of mics will meet those conditions at a time.

Also note that your approach for choosing which angle to use for $b$ only works if you consider the whole array as a single point (the distance between the mics relative to the distance to the source is vanishingly small). Otherwise, if the source is between the middle of side $a$ and one of $a$'s mics on the same axis as side $a$, angle $b$ will point the wrong way.



Instead of doing it like that, you might look at solutions using more than two mics at once.

An approach recently suggested here uses three equidistant mics and takes the cross product of the vectors pointing from one mic to another, with vectors having an additional component for the time difference.

Here's a really simple solution based on that, using axis-aligned diagonal mic pairs. Set your mics up like this:

      4
    1   2
      3


The time delta between mics 1 and 2 is the X component of a vector towards the source, and the time delta between mics 3 and 4 is the Y component. Check it out here.