What determines the maximum ground speed of hot air balloons?
Balloons travel with the wind, so all their motion over ground comes from air movements.

do hot air balloons have the ability to control their ground speed? If so, how?

This was a burning question in the nineteenth century and many attempts were made to influence ground speed, apart from rising into higher altitudes and testing whether the wind direction there is different. Naive people added sails but had to notice that those remained limp and only added weight. Others added paddles or engines (steam, electric and gasoline engines were tried) but success was limited before Zeppelins started to make directed lighter-than-air travel routine.
An interesting variety was tried by the Swedish balloonist Salomon August Andrée who added three long ropes to his balloon, their ends dragging on the ground. This caused drag which slowed the balloon, thus giving it some speed difference versus the wind. Now, with some relative wind, sails could indeed generate a force which could move the balloon sideways. However, when tried in 1897, two of the ropes were lost during launch and the remaining one proved ineffective.

Andrèe's balloon (picture source)

Is there a maximum limit to the ground speed of a hot air balloon?

No, if the balloon rises into the jet stream, it can travel at several hundred miles per hour. There is no speed limit once the balloon is up in the air.
However, filling and launching the balloon requires calm winds; hot air balloons almost never start in winds higher than 10 miler per hour. Landing in higher winds becomes dangerous, so here also a calm wind is preferred.

What factors contribute to this speed limit?

This is not about ground speed limits, but wind speed limits: While there is no fixed limit, higher winds will prevent the envelope to stretch out completely and rise up when being prepared for launch. Instead, the windward side will remain dented. During landing, the gondola will be dragged along on the ground when wind is too high. Both these factors let balloonists prefer calmer wind for take-off and landing. Once the balloon is up in the air, no such limit exists any more.