Measure rpm of a DC motor using two timers/counter and IR sensor (atmega 328p)
Putting the software listing aside for the moment, it makes me wonder if your opto-sensor circuit is producing lots of fast-switching edges as it the photo-transistor beam is broken or made. Can you look closely at those edges on an oscilloscope.

Looking at the sensor, there's no schematic given for it but it says it uses a LM393 dual comparator and I can see 6 resistors. That means they could have a Schmitt trigger on the photo-transistor output to prevent the output multi-switching with slow beam interruptions, when the photo-transistor output is going on or off slowly.

Also, LM393 is an open-collector part but it's almost a given that there's an output pull-up resistor on board there. Almost...but do try putting a 4K7 pull-up between the board output and the supply rail (3V3 I imagine), see if it improves.