Clock drift - is the precision of the crystal the only culprit?
The problem is the interaction between the interrupt handler that updates the value for the millis() function and the neopixels show() function.

To maintain the counter for the millis() function a interrupt is triggered approximately every 1.024 ms.

When the show() function is called all interrupts are disabled to get the precise timing needed to send the data to the neopixels.  For a 60 led strip this takes about 1800 microseconds (60 leds * 24 bits * 1.25 us per bit) so at least one interrupt is skipped each time you update the display.

This page has details about how the millis() function works.