Subtracting One Frequency From Another
You may be over-driving the local oscillator (pin 6) input to SA612. Data sheet says:


  External LO should be 200 mV (peak-to-peak) minimum up to 300 mV (peak-to-peak) maximum.


A square wave local oscillator coupled into pin 6 via a 1nF capacitor should be acceptable.

You are certainly over-driving mixer inputs (pin 1 &amp; 2). With input power above -20 dBm, distortion becomes significant. Output amplitude can only swing plus-or-minus 0.6V. Input voltage should certainly be less than 60 mV. The unused input (pin 2?) should be grounded through a capacitor (0.1uF or thereabouts).

Expect to see lots of high-frequency "stuff" at mixer outputs. To properly see the desired 0 - 1kHz signal, a low-pass filter with cut-off frequency of about 1 kHz is required.



You ask for a simpler approach:
A XOR logic gate can function as a mixer (like SA612), or you can view it as a phase comparator. One input connects to a 40 kHz fixed-frequency oscillator, while the other input connects to an amplified 40 kHz receiver. Both inputs should be logic-level square waves. Followed by an analog low-pass filter, you should see voltage wobble as the phase of your receiver's input transducer varies.