Training multi-label classifier with low quality training set So I'm creating a topics classifier where a document may be tagged for several different topics, let's say - A, B while actually the document belongs to A, B and C. 
In the training stage I want the classifier to learn that the document belongs to A and B but I'm not sure about class C so I don't want it to learn that the document doesn't\does belong to class C. 
Any ideas on how to implement such thing? 

I thought about adding weights to the output labels (low weight means that there's no way the document belongs to the aspect, high weight means that the document belongs to the aspect for sure and mid weight means I'm not sure (so the penalty in this case will be lower).
You almost solved the problem in the last paragraph. Expressed more formally, your cost function could be

$$\frac{1}{N} \sum_{i,j} c_{i,j} y_{i,j} \log x_{i,j}$$ 

where $i$ runs over items/documents, and $j$ runs over classes, $x$ is your prediction, $y$ is the binary label (1 if item $i$ has class $j$), and $0 &lt; c &lt; 1$ is your confidence. This is a simple modification of the cross entropy. When the confidence $c$ is low, the value of the prediction matters less.