What determines the coefficients in the wavefunction of a hybrid orbital?
For the $sp$ case, the other wavefunctions you listed would be equally valid. However, these are just $sp_+$ and $sp_-$ with the signs flipped. Since we square wavefunctions to get the probability density, these would give the exact same physical observables.

As for $sp^2$, the reason for the particular choice of $\alpha,\beta,\text{ and }\gamma$ comes precisely from the name of this hybridization: we want an orbital that is 2 parts p orbital for every 1 part s orbital. In other words, we want a mixture that is 1/3 s and 2/3 p which happens with exactly the normalization coefficients you listed. This is a second constraint. Now this still will lead to an infinite number of different solutions, as we can consider rotating these orbitals by any amount in the xy plane, which would give another valid solution. One of the nice things about the choice you listed is that one of the MOs is aligned with the y-axis and the other two are mirrored across the y-axis, so it is easy to consider the effect of axis symmetry operations on this choice of orbitals. If we include this third constraint of having one of the MOs aligned with an axis, we get a unique solution (besides possible sign flips and that we could align the orbital along the x axis instead) for $sp^2$.