Working with audio data with different sample rates in Tensorflow I am trying to implement (as a toy project) some aspects of speech recognition in Tensorflow. The audio files I want to use as training and test data have different sample rates (16, 20, 44 and 44.1 kHz).

The following list is not exhaustive, just listing a few options I could think of.


I can resample the whole signal (which is quite un-periodic) using scipy.signal.resample, probably after padding with zeros to achieve a signal length which is a power of 2, chopping of the equivalent amount afterwards.
I can try to chop the signal into short, non-overlapping windows, which I modify using scipy.signal.resample before glueing them back together and applying the rest of the chain.
I can chop the signal up into overlapping windows, scipy.signal.resample them and stick them together in a weighted fashion.
I can take the spectrogram tensor obtained using tf.contrib.signal.stft and try to stretch it, for example using rational function interpolation (reflecting tube models of the vocal tract)
I can assume that the model will learn to abstract away from sample rates, maybe even helping it (or hindering?) by adding convolution layers before the actual model I am interested in.


In which step of the model should I normalize sampling rates, and how should I do it?
If you have ffmpeg installed, you can just use tf.contrib.ffmpeg.decode_audio. That function takes a samples_per_second argument and does the resampling for you. The following calculates a log magnitude short-time spectrogram of your data.

waveform = tf.squeeze(
    tf.contrib.ffmpeg.decode_audio(
        tf.read_file(tf.placeholder(tf.string, name="filename")),
        file_format=tf.placeholder(tf.string, name="format"),
        samples_per_second=p["sample_rate"],
        channel_count=1))

log_mag_spectrogram = tf.log(tf.abs(tf.contrib.signal.stft(
    waveform, win_length, hop_length,
    n_fft, pad_end=False) + 1e-8))