How do you balance an Arduino quadcopter using calculated gyro values?
That was happening with me too. Check if the weight distribution of the drone is perfect. Lets say if the battery is placed a little more to one side of the frame than the other side, your drone will try to hover from the other side. 
The center of gravity should be taken care of when the drone is not balancing properly.