Unexpected YAW angle from magnetometer
If the magnetometer is ideal (or the mag_x and mag_y signals are calibrated), and the earths magnetic field vector is the largest signal that the magnetometer is seeing, then yes I would expect the above equations to work.
If the mag_x and mag_y signals from the magnetometer are raw, it's likely that there is a gain and an offset in the signals that are intrinsic to most magnetometers that could be affecting your results. Most magnetometers look like this (for x and y):
\$ Actual Mag_x*gain_x + offset_x = Mag_x\$
\$ Actual Mag_y*gain_y + offset_y = Mag_y\$
If the gain and offset varies significantly the yaw value will be difficult to calculate. The problem is, we don't know the gains or offsets, to find them we need a calibration.
Magnetometers can be calibrated, once the magnetometer is calibrated, you should be able to feed in the calibrated channels into your equation and get a reasonable result.
Also make sure that you aren't in an area with magnetic field noise, there could be other sources that you may be picking up instead of the earths magnetic field. Also make sure you are rotating the magnetometer in the same axis (no translation). A magnetometer by itself can have issues if you're using this for navigation, typically a gyro, accelerator and magnetometer (and sometmies GPS) are paired in an IMU to sense location.