How does someone design a low frequency FM radio transmitter?
I've been told that using VCO circuits to simply generate a low
frequency signal and modulate it with an input signal would be
unstable and unreliable

There's no evidence to suggest that what you say is true.
I'd consider this as a good basis for a decent VCO at 80 kHz: -

Picture from this site. Or consider the LM567 from the same site: -

Then you'll need a tuned coil driver to implement magnetic field comms as you imply.

I'm still a beginner with RF engineering so any help would be valuable

It's a magnetic field transmitter so, make your coil as big as you can and use decent capacitors to parallel tune it. The Q of your tuned circuit can be higher if your data rate is lower but, if I were designing it, I would not hesitate to model it all in a simulator tool.

I'm not interested in transmitting voice links but rather simple
files.

Rather than call it FM, call it FSK - it stands for frequency shift keying and can use exactly the same circuits shown above.
You also have to bear in mind that this type of transmission and reception uses the magnetic part of an electromagnetic wave. This is because the electric field part will not penetrate very far through the strata due to mineralized water for one thing. Additionally, because the frequency is low, the &quot;antenna&quot; is very short for the wavelength used and, it would be useless at transmitting any serious E-field any way.
So, it uses a magnetic field and, as you move away from the &quot;antenna&quot; (basically a large diameter coil of wire), the magnetic field falls with distance cubed. Compare this with a regular RF system that can utilize both E and H-fields; those fields fall linearly with distance. This is the magical thing about proper radio - in open fields (for example), for the same power to the &quot;antenna&quot;, proper RF goes waaaaaaaaaay further than E or H-fields on their own. Bear this in mind.