How did chemists in the 18th century learn that magnesium gain mass when it's burned?
The phlogiston hypothesis is about a substance, phlogiston, which was supposed to exist within matter that be consumed or released when it is combusted.  If this were true, then when magnesium is combusted the mass would decrease.

To someone who believed this to be true, it would come as a great surprise to find that if they combusted 100g of magnesium the mass would increase by about 65%.  This would be easy to measure with a set of balances, even with much smaller masses of a few grams.

A layman might not come across many day-to-day examples of a combustion reaction with a mass increase, since most things that burn release a lot of mass as gases, and that effect is what phlogiston was intended to explain.  One possibility, aside from burning metal, is the oxidation of metal in air.  Someone could weigh an iron nail or other object, allow it to rust, and weigh again to show it has gained mass.  This could easily be demonstrated by any merchant with a set of scales, but it's not obvious the rust is an oxide related to the combustion products.