title: Calculating multi-label inter-annotator agreement in Python
body: <p>Can anyone recommend a particular metric/python library for assessing the agreement between 3 annotators when the data can be assigned a combination of labels (as seen below)?</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th></th>
  <th>Msg_1</th>
  <th>Msg_2</th>
  <th>Msg_3</th>
  <th>Msg_4</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Annotator_1</td>
  <td>a,b,c</td>
  <td>b</td>
  <td>c</td>
  <td>a,b,c</td>
  </tr>
  <tr>
  <td>Annotator_2</td>
  <td>a,c</td>
  <td>b</td>
  <td>c</td>
  <td>a</td>
  </tr>
  <tr>
  <td>Annotator_3</td>
  <td>b,c</td>
  <td>a,b</td>
  <td>c</td>
  <td>a,b</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>I have tried python Krippendorff's Alpha implementations, however they don't seem to work with multilabels.</p>
  <p>Thanks.</p>