Attiny84 and 2-Axis Joystick
While @alexan_e’s suggestion fixed your practical problem, I would argue that it did not address your root cause, which is that whatever board setting you’re working from in boards.txt assumes a clock rate of 8MHz, while in reality your fuses are set to boot the microcontroller at 1MHz. This caused your delay(100) to take 800ms, and was the entire reason for the slowdown. 

Instead of fixing this at runtime, as aleman_e suggested, it seems more natural to me to bring your fuse settings into alignment with your compiler settings, using the “burn boot loader” Arduino command (which on Attinys generally does NOT burn a boot loader, but only sets the fuses to be in sync with the compiler settings).

If your application is not going to become much more complex, running at 1MHz might be a more efficient choice than running at 8MHz.