When to use a Hall probe and when a fluxgate magnetometer to measure magnetic fields?
I don't know the threshold for where you could use either, but a Hall Sensor is used for much higher current and detects magnetic force from the Hall Effect while some magnetometers can detect a magnetic field about 4 orders of magnitude lower than the earth's magnetic field.
Small Hall sensors are very inexpensive in analog or digital forms, but where accuracy is needed a magnetometer is used. The magnetometer can also be used to sense &gt;200A with high accuracy and linearity and requires the ferrite loop coils and IC.
The Hall Sensor can be very high speed up to 1MHz.
Unless you narrow your scope with specifications and budget , the choices are too broad.
e,g FG mag.  IC
$4  DRV421  Offset and drift : ±8 µT max    2% accuracy 50 ppm/'C  -40~+125'C

Always define your requirements 1st, Min/max ( H or I) for sensitivity, accuracy, cost, then add more specs as they become apparent.