title: Keras: Big one-hot-encoding: binary_crossentropy or categorical_crossentropy
body: <p>I'm training a text classification model where the input data consists of 4096 term frequency–inverse document frequencies.</p>
  
  <p>My output are 416 possible categories. Each piece of data has 3 categories, so there are 3 ones in an array of 413 zeros (one-hot-encodings)</p>
  
  <p>My model looks like this:</p>
  
  
  
  <pre class="lang-python prettyprint-override"><code>model = Sequential()
  model.add(Dense(2048, activation="relu", input_dim=X.shape[1]))
  model.add(Dense(512, activation="relu"))
  model.add(Dense(416, activation="sigmoid"))
  </code></pre>
  
  <p>When I train it with the <code>binary_crossentropy</code> loss, it has a loss of 0.185 and an accuracy of 96% after one epoch. After 5 epochs, the loss is at 0.037 and the accuracy at 99.3%. I guess this is wrong, since there are a lot of 0s in my labels, which it classifies correctly.</p>
  
  <p>When I train it with the <code>categorical_crossentropy</code> loss, it has a loss of 15.0 and an accuracy of below 5% in the first few epochs, before it gets stuck at a loss of 5.0 and an accuracy of 12% after several (over 50) epochs.</p>
  
  <p>Which one of those would be right for my situation (large one-hot-encodings with multiple 1s)? What do these scores tell me?</p>
  
  <p>EDIT: These are the <code>model.compile()</code> statement:</p>
  
  <pre class="lang-python prettyprint-override"><code>model.compile(loss='categorical_crossentropy',
                optimizer=keras.optimizers.Adam(),
                metrics=['accuracy'])
  </code></pre>
  
  <p>and</p>
  
  <pre class="lang-python prettyprint-override"><code>model.compile(loss='binary_crossentropy',
                optimizer=keras.optimizers.Adam(),
                metrics=['accuracy'])
  </code></pre>