ATtiny44 millis() not working with 16 MHz external clock
Let me guess: "much more time" is about, ohhh, 16 times?

When millis() was written, it had to assume what the input clock was. There is no way to detect the speed of the input clock: to do so would require another clock! So the writers of the function defined a starting constant, set it to 1000000, and required anyone that changed the clock would have to change that constant.

Find the constant, and set it to 16000000. Voilà!

Here's an article on the source code of the function:

https://ucexperiment.wordpress.com/2012/03/16/examination-of-the-arduino-millis-function/