A captive balloon tends to drift down the wind and the harder the wind blows, the further the balloon drifts.
In low or gusty winds a kite can nose-dive, losing a large amount of height even if it recovers.