title: document classification using keras - hierarchical sequence of digits
body: <p>I'm currently approaching a classification problem with the following situation:</p>
  
  <p>The labels are always 5 digits long, e.g.:</p>
  
  <pre><code>99923 this is sample document one
  56743 this is sample document two
  ...
  </code></pre>
  
  <p>where the first single digit stands for a certain category, every following digit for a subcategory and so on. </p>
  
  <p>Currently I'm using Keras with the following settings:</p>
  
  <pre><code>model = Sequential()
  model.add(Dense(512, input_shape=(vocab_size,)))
  model.add(Activation('relu'))
  model.add(Dropout(0.3))
  model.add(Dense(512))
  model.add(Activation('relu'))
  model.add(Dropout(0.3))
  model.add(Dense(num_labels))
  model.add(Activation('softmax'))
  model.summary()
  model.compile(loss='categorical_crossentropy',
                optimizer='adam',
                metrics=['accuracy'])
  </code></pre>
  
  <p>Because my training data is limited (around 80k samples overall), I decided to use only the first digit to estimate the main-category and I got pretty good results with ~90% accuracy without any preprocessing which yet has to be done.</p>
  
  <pre><code>5 - this is sample of maincategory 5
  9 - this is sample of maincategory 9
  ...
  </code></pre>
  
  <p>Now I wanted to approach a level further and use two digits to predict the main-category and the first subcategory. That brought me to the problem that there is often not a single sample for e.g. the combination "12".</p>
  
  <pre><code>51 - this is sample of maincategory 51
  95 - this is sample of maincategory 95
  ...
  </code></pre>
  
  <p>I told Keras to only use the labels with at least 1 sample (knowing that this is crap) and got around 40 labels with an overall accuracy of 85% which seems to be pretty good given the fact that I've lost many samples.</p>
  
  <p><strong>My question is</strong>:</p>
  
  <p>Can this kind of prediction be done more easily/efficiently? If I let my "two-digits" model predict an unseen sample out of a category which hasn't been trained, I would run into the problem of fitting a sample into a wrong category...</p>
  
  <p>Can I solve this prediction problem using Keras?</p>