Noise issues when driving DC motor with PWM
That circuit looks dodgy. The MCP1700 has an absolute maximum rating of 6.5V, but the note says battery voltage could be as high as 6.8V! 

D3 should be a Schottky type if you use a reasonable PWM frequency - and it's not just for MOSFET protection, it recirculates current through the motor which reduces peak current and improves efficiency (provided PWM frequency is high enough).

At 75Hz the motor's inductance is not sufficient to smooth out the current flow, so it becomes a series of high current pulses which could induce interference via magnetic coupling, battery voltage variations or ground currents. Most small dc motors need at least 3kHz PWM, however the human ear is very sensitive to this frequency so a higher frequency may be quieter. 

Tape playback amplifiers generally have lowest amplification at 6~10KHz where the tape head produces highest output, so the circuit should be less sensitive to interference in this range. If that isn't sufficient then you could increase it to 15-20KHz. 


  I tried filter caps on the motor terminals, which did not help.


The motor should already have suppression caps on it. If it doesn't then a 0.01~0.1uF ceramic cap from each terminal to the case will be sufficient to suppress brush noise. Higher values are counter-productive because they cause current spikes as they are rapidly charged and discharged by the PWM pulses.