Did the ancients really think mangoes were garlic like?
Actually the 蒜 is evidence of a substrate influence from an Austroasiatic language (ancestor of Vietnamese xoài or Khmer ស្វាយ [svaay]) in this context. Indeed, this lexeme actually does have its own Chinese character, 檨, with Mandarin pronunciation shē (ㄕㄜ).
It is the Hakka and Min varieties which use this 檨 lexeme:

Taiwanese Min Nan: soāiⁿ (Peh-oe-ji), suāinn (Tai-lo), ㄙㄨㆮ˫ (Zhuyin)
Fuzhounese Min Dong: suông (Bang-ua-ce)
Taiwanese Hakka: sôn (Phak-fa-sü)

In many of these varieties, the pronunications of 蒜 are very close:

Taiwanese Min Nan: soàn (Peh-oe-ji)
Fuzhounese Min Dong: sáung (Bang-ua-ce)
Taiwanese Hakka: son (Phak-fa-sü)

Hence the substitution of 蒜 was motivated by phonetic similarity (假借) with the lack of &quot;availability&quot; of this 檨 character.
In many of these, the collocation 番檨, preferred in Min Dong and certain varieties of Hakka, may help distinguish it from just 蒜. Or it may just be that mangoes came a bit later to the northeastern Fujian area than the southern area.