Angle between North and East from magnetometer values
Your approach is sound, but the device with its 50-gauss dynamic range is not ideal for measuring fields this small.  It can still likely work with the proper correction.  The earth's magnetic field strength is at most about 600 milligauss and is not parallel with the earth's surface but is &quot;dipping&quot; at some angle, and your magnetometer has a specified offset of 60 milligauss even after the IIS2MDC's &quot;offset calibration.&quot;  This is more than enough to explain your anomaly.
To get a good reading take these steps.  Place the magnetometer on a flat, level surface.  Take your readings and record them.  Then repeat at each of your 90-degree positions (sounds like you have already done this).
For the &quot;x&quot; axis, look at any two readings that are 180 degrees apart.  One of these readings is the local magnetic field plus the offset, and the other is the negative of the local magnetic field plus the offset.  Add them together, and you have two times your offset, so divide by two.  You must subtract the offset from every x axis reading.  Repeat for two &quot;y&quot; readings 180 degrees apart for the &quot;y&quot; offset value.  Then you can do the &quot;z&quot; axis if you want by flipping the board over, although as you point out, &quot;z&quot; is not needed if you are always keeping the board level.  This will remove the inherent offset.  Be careful to keep the magnetometer in the same spot as you rotate to prevent any effect from local magnetic field variance, or be sure you are in an area free from iron and DC currents.  Applying these offset corrections will greatly improve your results.
Next, take two readings of &quot;x&quot; 90 degrees apart and perform the square root of the sum of the squares (after subtracting your offset value calculated above).  Do the same for &quot;y.&quot;  This provides the amplitude of the field and these numbers should theoretically be equal.  If they are not, there is a difference in sensitivity (gain) between the x and y values.  Apply a correction factor to one or the other to make them equivalent - it doesn't matter which one, since you are going to be taking a ratio of the two values.
Do not initiate another hardware &quot;offset calibration&quot; or you will have to repeat this process.  If you calibrate &quot;z&quot; as well, and add an accelerometer you can do some math and have a board that works even when it is not level; otherwise you need to make sure you are level and only need &quot;x&quot; and &quot;y.&quot;
Good luck!