Objects with negative mass?
Another solution using RigidBodies, Zero gravity and a Texture Field:

Note that all the balloons here are single user object instances all with RigidBodies.


turn RigidBodies World Gravity to 0.0
Add a texture field which uses RGB values of the texture to determine the force (using a Color Ramp). R or G or B values of 0.5 correspond respectively to 0.0 force in the X, Y or Z axis. 0.0 R, G or B values correspond to +1.0 force and 1.0 R, G, B values correspond to -1.0 X,Y or Z force. Its means that a downward Z axis force of -1.0 corresponds to R=0.5,R=0.5,B=1.0, and an upward Z axis force or 1.0 is R=0.5,G=0.5,B=0.0. Thus the blueish/yellowish texture below. See https://docs.blender.org/manual/en/latest/physics/forces/force_fields/types/texture.html for details


Also note that I am using a Clouds texture here which is 3-dimensional which explains why the balloons are sometimes switching direction, when they pass through a 3D space where the 3D texture is changing color. May be using an image texture or tweaking how the texture is build would prevent that.