What causes a HVAC control transformer to overheat?
Underpowered transformer
Sounds like you may not have gotten an equivalent replacement.  When looking for a replacement transformer, you should be concerned with

Input, or primary voltage
Output, or secondary voltage
Volt-amperes (VA) rating.

If the transformer is not designed to supply the amount of current needed by the components on the circuit, it will overheat and likely die. Current generates heat. The more current that flows through the transformers coil, the hotter those tiny wires get. Get the wires too hot, and they will break. Heating and cooling cycles of the transformer can cause the wires to become brittle, which can also lead to failure over time.
If you've recently added components to the system that draw power from the transformer (WiFi thermostat, electronic dampers, etc.), you may have to upgrade to a transformer with a higher VA rating. If you replaced an old transformer with a new one that had a lower VA rating, that could explain the overheating.
Short on transformer secondary
A short somewhere in the circuit on the secondary side of the transformer, could also lead to overheating.  Often times the secondary circuit has no overcurrent protection (fuse), so it can be easy to overload the circuit and not know until it's too late.