audio spectrogram normalization
First of all, yes, what can be achieved by any machine learning tool is only as good as the dataset it was trained on.

The ML can't guess what properties of the high-dimensional data you want it to respect and which not.

So, you'll have to make sure all your data sources are either equal in their probability distributions when it comes to the feature you're interested in (language), essentially. 

A relatively easy way of doing so is random sampling, where you categorize all data sources according to languages, and then for each language, pick a subset of recordings so that all sources are equally frequent.  You then train with different randomly picked subsets.

Really, this is Bayesian statistics basics; I don't think signal processing will help you here. You can sure still try to preprocess the audio data so that it only consists of relatively recording-method-invariate features (i.e. classical audio features, e.g. Cepstrum coefficients), but then you'd basically be doing what you'd instead wanted to use a neural net for: finding appropriate properties of the signal from which to tell the languages apart, and since the problem would persist, I'm pretty optimistic that the Resnet approach would still large correlate its classification to the data source, since some information would certainly persist through that feature-finding step, and your datasets still aren't even in terms of classification target pdf.