Comparing audio waveform in a visual and / or numeric way
You could try a SoX and imagemagick solution, which could visually diff spectrograms, for example:

# generate spectrograms with SoX
$ sox 0000001353.noframedrop_Lg.small_1_edit.ogg -n spectrogram -arl -o spectrogram_1.png
$ sox 0000001353.noframedrop_Lg.small_2_edit.ogg -n spectrogram -arl -o spectrogram_2.png

# imagemagick
$ composite spectrogram_1.png spectrogram_2.png -compose difference spectrograms_difference.png


This could give you a quick heads up if there is a difference, similar to how you do for video streams, however is not suited to precisely locating TC positions. Below is an example with one of two source SoX spectrograms generated above (10min speech), and its imagemagick comparison with the other (nonblack indicates a difference):