Creating a valid dataset for obtaining results I have created a domain-specific dataset, lets say it is relating to python programming topic posts. I have taken data from various places specific to this topic to create positive examples in my dataset. For example, python related subreddits, stack exchange posts tagged with python, twitter posts hashtagged with python or python specific sites.
The data points taken from these places are considered positive data points and then I have retrieved data points from the same sources but relating to general topics, searched if they contain the word python in them and if they do discard them to create the negative examples in my dataset.
I have been told that I can use the training set from the dataset as is, but that I need to manually annotate the test set for the results to be valid, otherwise they would be biased. Is this correct? How would they be biased? To be clear the test set contains different entries to the training set.
There are close to 200,000 entries in the test set which makes manual annotation difficult. I have seen similar methods been used in papers I have previously read without mention of manual annotation. Is this technique valid or do I have to take some extra steps to ensure the validity of the test sets?
There are two potential biases:

With this automatic method, you might have a few erroneous labels. For example it happens regularly here on DataScienceSE that a user tags a question &quot;python&quot; but actually the question is not specific to python at all. Same thing for the opposite case: for example it's possible that some content contains some python code but doesn't mention &quot;python&quot; anywhere.
The distribution between positive and negative classes is arbitrary. Let's assume you use 50% positive / 50% negative: if later you want to apply your classifier on a new data science website where only 10% of the content is about python, it's likely to predict a lot of false positive cases so the true performance on this data will be much lower than on your test set.

It's rare to have a perfect dataset, so realistically in my opinion the first issue is probably acceptable because the noise in the labels should be very limited. The second issue could be a bit more serious but this depends on what is the end application. Keep in mind that a trained model is meant to be applied to the same kind of data as it was trained/tested on.