Text annotating process, quality vs quantity? I have a question regarding annotating text data for classification.

Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time to go through all the text samples, but at least a significant portion of them.


Should we focus on generating new samples for each annotator? (They never see the same text samples as any other annotator) (quantity approach).
Or should all annotators see the same samples and the annotator agreement is taken to account? (quality approach).


Thoughts,


will generate more unique samples than 2. (more training samples for a classifier) - and hoping that in the feature extraction part, the useful features will appear by themselves.
will generate less unique samples, but with the annotator agreement taken into account. (less training samples for a classifier, but with higher quality)
Both metods are against the nature of machine/statistical learning. Lack of data will not let you generalize well in any case. It will have different effect though. Generally speaking:


You will be able to classify more test samples, but sometimes will do
it wrong due to errors in data. 
You will have precise classification of some test samples, but will have no idea how to classify the rest of them.


So, it's up to your's classification score/quality estimation function which case is better for you.