How can I make this hexagon be exactly the height of the square
By splitting a hex of height 1 into equilateral triangles, and Pythagoras, we know its side-to-side width is sqrt(3)/2. So, if we just look at the threshold in X, we know that has to be sqrt(3)/4. (And, IMO, that's an easier way to think about it than considering the dot-product branch of your tree)

You would hard-code that value into the Less Than, though, so it doesn't have to be recalculated for every shading-point... (the same goes for your existing sqrt(3). )