Training multi-label classifier with low quality training set
You almost solved the problem in the last paragraph. Expressed more formally, your cost function could be

$$\frac{1}{N} \sum_{i,j} c_{i,j} y_{i,j} \log x_{i,j}$$ 

where $i$ runs over items/documents, and $j$ runs over classes, $x$ is your prediction, $y$ is the binary label (1 if item $i$ has class $j$), and $0 &lt; c &lt; 1$ is your confidence. This is a simple modification of the cross entropy. When the confidence $c$ is low, the value of the prediction matters less.