title: Keras: what does class_weight actually try to balance?
body: <p>My data has extreme class imbalance. About 99.99% of samples are negatives; the positives are (roughly) equally divided among three other classes. I think the models I'm training are just predicting the majority class basically all the time. For this reason, I'm trying to weight the classes. </p>
  
  <p><strong>Model</strong></p>
  
  <pre><code>model = Sequential()
  
  #Layer 1
  model.add(Conv1D( {{choice([32, 64, 90, 128])}}, {{choice([3, 4, 5, 6, 8])}}, activation='relu', kernel_initializer=kernel_initializer, input_shape=X_train.shape[1:]))
  model.add(BatchNormalization())
  
  #Layer 2
  model.add(Conv1D({{choice([32, 64, 90, 128])}}, {{choice([3, 4, 5, 6])}}, activation='relu',kernel_initializer=kernel_initializer))
  model.add(Dropout({{uniform(0, 0.9)}}))
  
  #Flatten
  model.add(Flatten())
  
  #Output
  model.add(Dense(4, activation='softmax'))
  </code></pre>
  
  <p>(The <code>{{...}}</code> are for use with <a href="https://github.com/maxpumperla/hyperas" rel="noreferrer">Hyperas</a>.)</p>
  
  <p><strong>How I've tried to weight it</strong></p>
  
  <p>\1. Using <code>class_weight</code> in <code>model.fit()</code></p>
  
  <pre><code>model.fit(X_train, Y_train, batch_size=64, epochs=10, verbose=2, validation_data=(X_test, Y_test), class_weight={0: 9999, 1:9999, 2: 9999, 3:1})
  </code></pre>
  
  <p>\2. Using <code>class_weight</code> in <code>model.fit()</code> with <code>sklearn</code> <code>compute_class_weight()</code></p>
  
  <pre><code>model.fit(..., class_weight=class_weight.compute_class_weight("balanced", np.unique(Y_train), Y_train)
  </code></pre>
  
  <p>\3. With a custom loss function</p>
  
  <pre><code>from keras import backend as K
  def custom_loss(weights):
      #gist.github.com/wassname/ce364fddfc8a025bfab4348cf5de852d
  
      def loss(Y_true, Y_pred):
          Y_pred /= K.sum(Y_pred, axis=-1, keepdims=True)
          Y_pred = K.clip(Y_pred, K.epsilon(), 1 - K.epsilon())
  
          loss = Y_true * K.log(Y_pred) * weights
          loss = -K.sum(loss, -1)
          return loss
  
      return loss
  
  extreme_weights = np.array([9999, 9999, 9999, 1])
  model.compile(loss=custom_loss(extreme_weights),
              metrics=['accuracy'],
              optimizer={{choice(['rmsprop', 'adam', 'sgd','Adagrad','Adadelta'])}}
              )
  
  #(then fit *without* class_weight)
  </code></pre>
  
  <p><strong>Results</strong></p>
  
  <p>Poor. Accuracy across all classes is ~<code>.99</code>, and unbalanced accuracy for all classes is  ~<code>.5</code>. But more meaningful metrics, like auPRC, tell a different story. The auPRC is nearly <code>1</code> for the majority class, and nearly <code>0</code> for the rest. </p>
  
  <p>Is this how Keras balances classes? It just makes sure that the accuracy is the same across them—or should either metrics be equal or comparable too? Or am I specifying the weights wrong?</p>