title: How to calculate accuracy for Multiclass - Multilabel classification?
body: <p>I am working on a multilabel and multiclass classification framework, I want to add matrices for multilabel and multiclass accuracy calculation.</p>
  <p>Here is demo data :</p>
  <pre><code>predicted_labels = [[1,0,0,0,1],[1,0,0,0,1],[1,0,0,0,1],[1,0,0,0,1],[1,0,0,0,1],[1,0,1,0,1]]
  true_labels      = [[1,1,0,0,1],[1,0,0,1,1],[1,0,0,0,1],[1,1,1,0,1],[1,0,0,0,1],[1,0,0,0,1]]
  </code></pre>
  <p>Most popular accuracy matrices for multi-label, multi-class classification are :</p>
  <ol>
  <li>Hamming score</li>
  <li>Hamming loss</li>
  <li>Subset accuracy</li>
  </ol>
  <p>The code for the above three is :</p>
  <pre><code>def hamming_score(y_true, y_pred, normalize=True, sample_weight=None):
      '''
      Compute the Hamming score (a.k.a. label-based accuracy) for the multi-label case
      
      '''
      acc_list = []
      for i in range(y_true.shape[0]):
          set_true = set( np.where(y_true[i])[0] )
          set_pred = set( np.where(y_pred[i])[0] )
          #print('\nset_true: {0}'.format(set_true))
          #print('set_pred: {0}'.format(set_pred))
          tmp_a = None
          if len(set_true) == 0 and len(set_pred) == 0:
              tmp_a = 1
          else:
              tmp_a = len(set_true.intersection(set_pred))/\
                      float( len(set_true.union(set_pred)) )
          #print('tmp_a: {0}'.format(tmp_a))
          acc_list.append(tmp_a)
          
      return  { 'hamming_score' : np.mean(acc_list) , 
                'subset_accuracy' : sklearn.metrics.accuracy_score(y_true, y_pred, normalize=True, sample_weight=None), 
                'hamming_loss' : sklearn.metrics.hamming_loss(y_true, y_pred)}
  </code></pre>
  <p>But I was looking for f1-score for multilabel classification so I tried to use sklearn f1-score :</p>
  <pre><code>print(f1_score(demo, true, average='micro'))
  </code></pre>
  <p>But it gave me the error :</p>
  <pre><code>ValueError: multiclass-multioutput is not supported
  </code></pre>
  <p>I converted the data into np array and use f1_score again:</p>
  <pre><code>print(f1_score(np.array(true_labels),np.array(predicted_labels), average='micro'))
  </code></pre>
  <p>Then I am getting the accuracy :</p>
  <pre><code>0.8275862068965517
  </code></pre>
  <p>I tried one more experiment, I used one-one example from true and predicted labels and used f1-score over that and then took the mean of that :</p>
  <pre><code>accuracy_score = []
  
  for tru,pred in zip (true_labels, predicted_labels):
      accuracy_score.append(f1_score(tru,pred,average='micro'))
      
  print(np.mean(accuracy_score))
  </code></pre>
  <p>output:</p>
  <pre><code>0.8333333333333335
  </code></pre>
  <p>Accuracy is different</p>
  <p>Why it's not working on list of list but working on np array and which method is correct, taking one by one example and mean or using numpy array with all samples?</p>
  <p>What other matrices are available for multilabel classification accuracy calculation?</p>