Atmel SAMC21 MCU internal oscillator based counter inaccurate
You're actually taking an interrupt every 100 &micro;s, not millisecond as your naming implies. That's a lot of interrupts for a 48 MHz system.

Furthermore, the code hints at a rather elaborate interrupt handling structure that exists "behind the scenes", with callbacks being "registered" and "enabled". So while your actual callback function is short and sweet, the overall overhead for taking any individual interrupt may still be rather high.

You also hint at using a UART for communication, and I'm guessing that its driver is also interrupt-based.

All of this leads me to suggest that you are approaching the limit on the number of interrupts that this system can handle, and that some of your timer interrupts are simply being missed. This would be especially true if the UART interrupts have a higher priority than the timer interrupts, which is the usual case. Try increasing the timer interrupt period to 200 &micro;s or 1000 &micro;s. (In your callback, increment the millis variable by 2 or 10, respectively &mdash; then the rest of the system won't know the difference.)