Tracking 2D positioning with IMU Sensor
Non-zero rates are normal for MEMS accelerometers and gyros. This is persistent error. It is eliminated by somehow making sure that the device is stationary for a couple of seconds (so the output can stabilize), then getting a reading. Henceforth, subtracting this steady-state error from all future measurements. Look up the datasheet of your sensor - there will be maximum values for this and other types of measurement tolerances.

Now, the much more complex subject of fusing the accelerometer, gyro and compass data. This can get hugely complicated, using Kalman filter, like Apolo once did. It can, however, be quite simple as well.

The general idea is that the magnetic sensor has slow response, low accuracy, but the error does not increase. On the other hand, a gyro's output is velocity, which is integrated to get angular position. The error grows very fast - generally you can't do dead reckoning for more than a minute with only a giro. The accelerometer is worse - it outputs acceleration, which gets integrated twice!

So, a simple fusing filter would be some linear combination of the readings of the accelerometer and compass, with the coefficient in front of the gyro descreasing over tyme.

Here is a discussion by much more knowledgeable people than me on the topic. 

Note: What you are trying to do is called dead reckoning.