Does object mass have a significant bearing on wind interference?
Scaling laws are your friend here.

While the area which can be attacked by a gust grows with the square of the length increase, the mass of the drone will grow with the cube of length. This translates into higher disc loading of the props, which in turn requires them to operate at a higher dynamic pressure. The same wind gust will change forces relatively less on a bigger drone. It is the same with aircraft: Light indoor models have the same problems outside that you experienced with the small drone.

It gets even better when you look at the capability to resist rotation: Here the moment of inertia is the deciding factor which grows with the product of mass and length squared. This means the bigger drone will better resist attitude changes by the length increase to the fifth power. To be fair, gust moments will increase with the cube of length increase (area times lever arm), leaving you still with an advantage of length squared. This also means that all changes will happen more slowly, which improves the chances that the control system (the MEMS sensors onboard as well as your control inputs) will keep the situation under control.