Prescaled Timers Problem
The default clock source of the microcontroller is internal RC set to 8MHz.
The device also comes with the CKDIV8 fuse programmed which divides the clock by 8 and results to 1MHz that seems to match what you describe.

A simple test you can do is remove the crystal and power the circuit, if the mcu works then you are using the internal RC so you need to change the fuses to use the crystal as the clock source. If not then it's probably only related to the clock divider.

Regarding the clock divider please read my reply here