title: Tackling Class Imbalance: scaling contribution to loss and sgd
body: <p><strong>(An update to this question has been added.)</strong></p>
  
  <p>I am a graduate student at the university of Ghent, Belgium; my research is about emotion recognition with deep convolutional neural networks. I'm using the <a href="http://caffe.berkeleyvision.org/" rel="noreferrer">Caffe</a> framework to implement the CNNs.</p>
  
  <p>Recently I've run into a problem concerning class imbalance. I'm using 9216 training samples, approx. 5% are labeled positively (1), the remaining samples are labeled negatively (0).</p>
  
  <p>I'm using the <a href="http://caffe.berkeleyvision.org/doxygen/classcaffe_1_1SigmoidCrossEntropyLossLayer.html" rel="noreferrer">SigmoidCrossEntropyLoss</a> layer to calculate the loss. When training, the loss decreases and the accuracy is extremely high after even a few epochs. This is due to the imbalance: the network simply always predicts negative (0). <em>(Precision and recall are both zero, backing this claim)</em></p>
  
  <p>To solve this problem, I would like to <strong>scale the contribution to the loss depending on the prediction-truth combination</strong> (punish false negatives severely). My mentor/coach has also advised me to <strong>use a scale factor when backpropagating</strong> through stochastic gradient descent (sgd): the factor would be correlated to the imbalance in the batch. A batch containing only negative samples would not update the weights at all.</p>
  
  <p><em>I have only added one custom-made layer to Caffe: to report other metrics such as precision and recall. My experience with Caffe code is limited but I have a lot of expertise writing C++ code.</em></p>
  
  <hr>
  
  <p><strong>Could anyone help me or point me in the right direction on how to adjust the <a href="http://caffe.berkeleyvision.org/doxygen/classcaffe_1_1SigmoidCrossEntropyLossLayer.html" rel="noreferrer">SigmoidCrossEntropyLoss</a> and <a href="http://caffe.berkeleyvision.org/doxygen/classcaffe_1_1SigmoidLayer.html" rel="noreferrer">Sigmoid</a> layers to accomodate the following changes:</strong></p>
  
  <ol>
  <li>adjust the contribution of a sample to the total loss depending on the prediction-truth combination (true positive, false positive, true negative, false negative).</li>
  <li>scale the weight update performed by stochastic gradient descent depending on the imbalance in the batch (negatives vs. positives).</li>
  </ol>
  
  <p>Thanks in advance!</p>
  
  <hr>
  
  <h2>Update</h2>
  
  <p>I have incorporated the <strong><a href="http://caffe.berkeleyvision.org/doxygen/classcaffe_1_1InfogainLossLayer.html#details" rel="noreferrer">InfogainLossLayer</a> as suggested by <a href="https://stackoverflow.com/a/30497907/1714410">Shai</a></strong>. I've also added another custom layer that builds the infogain matrix <code>H</code> based on the imbalance in the current batch.</p>
  
  <p>Currently, the matrix is configured as follows:</p>
  
  <pre><code>H(i, j) = 0          if i != j
  H(i, j) = 1 - f(i)   if i == j (with f(i) = the frequency of class i in the batch)
  </code></pre>
  
  <p><em>I'm planning on experimenting with different configurations for the matrix in the future.</em></p>
  
  <p>I have tested this on a 10:1 imbalance. The results have shown that the network is learning useful things now: <em>(results after 30 epochs)</em></p>
  
  <ul>
  <li>Accuracy is approx. ~70% (down from ~97%);</li>
  <li>Precision is approx. ~20% (up from 0%);</li>
  <li>Recall is approx. ~60% (up from 0%).</li>
  </ul>
  
  <p>These numbers were reached at around 20 epochs and didn't change significantly after that.</p>
  
  <p><em>!! The results stated above are merely a proof of concept, they were obtained by training a simple network on a 10:1 imbalanced dataset. !!</em></p>