Training data from different sources
It seems that you have a domain adaptation problem. The samples from the two sources behaves differently.

I suggest reading Frustratingly Easy Domain Adaptation. As the name hints, the solution is easy , popular (800 citation until now) and a good survey of other directions.

I understand that the classifier that you run on the entire dataset was train on it. 
How well does the classifiers trained on the single sources perform on the other sources?
How many of the samples belong to the first source?
Will you have an indication at production of the source of the sample?
The answer to these question might open more directions.