Stopping drift in hovering quadrocopter
I think that you will want to use a combination of sensors.  The accelerometers and gyroscopes will be able to correct for strong breezes.  You can then use GPS to counter out the longer-term drift (or bias, as it is sometimes called).  I think that the combination of these two sensors in some sort of filter (probably a Kalman) will keep your position drift minimized.

GPS won't be accurate enough by itself though.  An alternative approach to the Kalman filter (which can get a bit math heavy) is to use the DCM algorithm from DIYdrones.  There seems to be a lot of success in using this so far.

Finally, the Parrot drone quadrotor uses a downward-facing 60fps camera to cancel out drift.  It looks down and extracts features from the ground under it, then uses a type of visual odometry (I'm assuming some sort of optical flow algorithm) to determine how far the quadrotor has drifted.  I believe that this only works at low altitudes on the Parrot, but I see no reason why it couldn't be extended to a higher altitude.