Arduino-controlled slot cars - alternative to MOSFETs for speed control?
YOu are entering a world where Electro Magnetic Compatibility is at risk. Pay attention to the stray current spikes that radiate noise (Egress) and their influence on other circuits of high impedance (Ingress). YOu probably also have ripple or the conducted egress and ingress as well. 

For radiated noise a handy tool is a small AM radio (not FM) nearby, to locate such noise issues if you dont have a scope. 

Noise suppression management may include some common mode chokes to the track or pot lines. Perhaps some supply rail filtering may need to be added around the track.   

I would use ferrite beads for each driver with caps after the bead and then use twisted pairs to send the current after going thru a ferrite torroid or similar common mode choke, like those used in Video cables. 
Its good to have a kit of parts for such issues.

Add decoupling caps to the breadboard for both sides.