Calculating error between two clocks?
What accuracy does the datasheet for your oscillator claim? Typically, a crystal oscillator will only guarantee to be within 25ppm.

Perhaps the act of measuring the frequency is having an effect also.