title: How to convert a softmax output to one-hot format in customized Keras loss
body: <p>In a <strong>2D semantic segmentation</strong> task. I want to calculate an average dice coefficient for each category in a customized Keras loss function.</p>
  
  <p>So I think <strong>the first step</strong> is calculate dice coefficients for each category, then <strong>average</strong> coefficients to get avg_dice.</p>
  
  <p>Now my loss function looks like</p>
  
  <pre><code>def avg_dice_coef(y_true, y_pred, n_classes, smooth=1e-5):
      # y_pred_new = K.variable(np_utils.to_categorical(K.argmax(y_pred), num_classes=OPTIONS.nb_classes))
  
      avg_dice = 0.  # 用于求和每个类别的骰子系数，之后求平均
      for class_index in range(n_classes):  # 对每个类别进行循环
          intersection = K.sum(y_true[:, :, :, class_index] * y_pred_new[:, :, :, class_index], axis=[1, 2])
          union = K.sum(y_true[:, :, :, class_index], axis=[1, 2]) + K.sum(y_pred_new[:, :, :, class_index], axis=[1, 2])
          dice_one_class = K.mean((2. * intersection + smooth) / (union + smooth), axis=0)
          avg_dice += dice_one_class
      return avg_dice / n_classes  # 之后求平均
  </code></pre>
  
  <p>in this function, y_pred is <strong>outputs from network after softmax</strong>, labels_shape=(batch_size, 1024, 512, n_classes), predicts_shape=(batch_size, 1024, 512, n_classes)</p>
  
  <p>I think my loss is wrong because I use float y_pred. According to the equation</p>
  
  <p><a href="https://i.stack.imgur.com/WoniP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/WoniP.png" alt="enter image description here"></a></p>
  
  <p>I think I should use integer 0 or 1 y_pred  value instead of float. So I need to 1) use K.argmax() to get the index of max value of each pixel, 2) convert the result of K.argmax() to one-hot format.(A simple example: convert [0.1, 0.1, 0.8] to [0, 0, 1])</p>
  
  <p>But when I add </p>
  
  <pre><code>y_pred_new = K.variable(np_utils.to_categorical(K.argmax(y_pred), num_classes=OPTIONS.nb_classes))
  </code></pre>
  
  <p>to achieve this goal, I got an error:</p>
  
  <p><strong>ValueError: setting an array element with a sequence.</strong></p>
  
  <p>How can I repair my loss and <strong>whether my idea of averaging is right?</strong></p>