LIME visualization outputs padded regions as important - Mel-spectrogram (audio analysis)
Feeding the neural network 75% irrelevant data (zero padding) seems less-than-ideal, both in terms of computational inefficiency and because it makes your model potentially vulnerable to this data.

I would consider a smaller model with 128x128 input input format (ex: MobileNet). You can either upscale the mel-spectrograms you have now from 128x64 -> 128x128 or extract windows that are 128 long (potentially adjusting FFT/hop size to make the window cover same area in time).