Is it advisable to merge similar datasets to improve model accuracy? I'm trying to build a classifier that would help me classify whether a statement collected from Reddit is bullish, bearish or neutral.
To this end, I have hand-labelled a fairly small dataset of 2500 entries, each with max 280 characters. Unfortunately, the data is unbalanced (60% neutral, 25% bullish and 15% bearish) and my initial attempts are returning poor results for the bullish and bearish classes.
I've managed to obtain 10,000 entries of similar data from StockTwits, each labelled bullish or bearish (no neutral...) and 280 characters max.
I tested supplementing my Reddit dataset with bullish/bearish data from this dataset to balance out the classes. In other words, I added data from the StockTwits dataset until the number of bullish and bearish entries in matched the neutral. With this model, I'm getting much better results.
Is merging similar datasets in this way advisable? My gut instinct says &quot;no&quot; but I haven't found anything suggesting this is not a good idea.
Ideally if you mix data from different sources (different distributions), the mix should be uniform for each class.   Otherwise if for example class A gets more of source S and class B gets more of source T, there is a risk that the model will learn to associate class A with source S and B with source T and that it won't generalize across sources.
To validate such an approach, keep a separate test set with both sources and compute several accuracies, one for each (class, source).