How is Gyro drift error related to error in velocity and position?
In a typical pure inertial nav solution, you must cancel out the acceleration due to gravity by subtracting it out.  If the vehicle's down vector is incorrect, this error appears as a lateral acceleration that is equal to \$\sin \theta_{error}\$ -- and for small angles is close to \$\theta_{error}\$ itself.  So if the gyro has an offset, \$\theta_{error}\$ grows linearly and therefore acceleration error grows linearly (at first).

Since velocity is the integral of acceleration, and position is the integral of velocity, a constant, linear acceleration drift (1st order polynomial) will result in a quadratic velocity error (2nd order polynomial) and cubic position error (3rd order polynomial).

If all you have is a 6-DOF IMU and a model of the earth's gravity, then both the IMU and the gravity model have to be perfect.