Choosing class labels from annotated data
That's one valid way of approaching the problem. In your final solution, though, it will be helpful to quantify the overall inter-rater agreement. For example, Cohen's kappa is a commonly-used metric:
\begin{eqnarray}
\kappa &amp;=&amp; \frac{p_{o}-p_{e}}{1-p_{e}}\\
       &amp;=&amp; 1 - \frac{1-p_{o}}{1-p_{e}},
\end{eqnarray}
where $p_{o}$ and $p_{e}$ are the amount of agreement you observe and that due to chance, respectively. The reason this is important is that the amount of agreement your human annotators achieve is a theoretical upper-bound on the performance of your machine learning solution—it provides context for interpreting the performance of your algorithmic approach.