How to shield for electromagnetic interference (DIY level)?
I'm not entirely surprised that you've got pickup. You've got wires hanging all over the place. I'd suggest two things to do:

1) Twist your phototransistor wires (blue and yellow) together to form a twisted pair. At the same time, make sure your phototransistor connections are as compact as you can without shorting anything - I suggest moving your connection pins much closer together. Also, make your 5k resistor connections using as short a pair of wires as possible - I recommend making the connection on the protoboard right at the Arduino pins without using any wires at all. As a bonus, get some copper braid and enclose your 20 cm pair in the braid, with one end tied to Arduino ground.

2) As close as possible to the Arduino, connect a small (100 pF to 1000 pF) capacitor between D2 and GND. I can't be more specific as to the value, since I don't know what rate the isolator will operate at. Generally, the bigger the better, except that at some point you'll find that the cap reduces signal amplitude as you rotate the motor shaft.