title: Keras - text classification, overfitting, and how to improve my model?
body: <p>i am developing a text classification neural network
  based on this two articles - <a href="https://github.com/jiegzhan/multi-class-text-classification-cnn-rnn" rel="nofollow noreferrer">https://github.com/jiegzhan/multi-class-text-classification-cnn-rnn</a>
  <a href="https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/" rel="nofollow noreferrer">https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/</a></p>
  
  <p>For the training i am using, text data in Russian language (language essentially doesn't matter,because text contains a lot of special professional terms, and sadly to employ existing word2vec won't be an option.) </p>
  
  <p>I have such parameters of training data - 
  Maximum lengths of an article - 969 words
  Size of vocabulary - 53886 
  Amount of labels - 12 (sadly they are distributed quite unevenly, for instance i have first label - and have around 5000 examples of this, and second  contains only 1500 examples.)</p>
  
  <p>Amount of training data set - Only  9876 entries. I'ts <strong>the biggest problem</strong>, because sadly i can't increase size of the training set by any means (only way out to wait another year☻, but even it will only make twice the size of training date, and even double amount is'not enough)</p>
  
  <p>Here is my code -</p>
  
  <pre><code>x, x_test, y, y_test = train_test_split(x_, y_, test_size=0.1)
  x_train, x_dev, y_train, y_dev = train_test_split(x, y, test_size=0.1)
  
  embedding_vecor_length = 100
  
  model = Sequential()
  model.add(Embedding(top_words, embedding_vecor_length, input_length=max_review_length))
  model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu'))
  model.add(MaxPooling1D(pool_size=2))
  model.add(keras.layers.Dropout(0.3))
  model.add(Conv1D(filters=32, kernel_size=4, padding='same', activation='relu'))
  model.add(MaxPooling1D(pool_size=2))
  model.add(keras.layers.Dropout(0.3))
  model.add(Conv1D(filters=32, kernel_size=5, padding='same', activation='relu'))
  model.add(MaxPooling1D(pool_size=2))
  model.add(keras.layers.Dropout(0.3))
  model.add(Conv1D(filters=32, kernel_size=7, padding='same', activation='relu'))
  model.add(MaxPooling1D(pool_size=2))
  model.add(keras.layers.Dropout(0.3))
  model.add(Conv1D(filters=32, kernel_size=9, padding='same', activation='relu'))
  model.add(MaxPooling1D(pool_size=2))
  model.add(keras.layers.Dropout(0.3))
  model.add(Conv1D(filters=32, kernel_size=12, padding='same', activation='relu'))
  model.add(MaxPooling1D(pool_size=2))
  model.add(keras.layers.Dropout(0.3))
  model.add(Conv1D(filters=32, kernel_size=15, padding='same', activation='relu'))
  model.add(MaxPooling1D(pool_size=2))
  model.add(keras.layers.Dropout(0.3))
  model.add(LSTM(200,dropout=0.3, recurrent_dropout=0.3))
  model.add(Dense(labels_count, activation='softmax'))
  model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
  
  print(model.summary())
  
  model.fit(x_train, y_train, epochs=25, batch_size=30)
  scores = model.evaluate(x_, y_)
  </code></pre>
  
  <p>I tried different parameters and it gets really high accuracy in training (up to 98%)
  But i really performs badly on test set. Maximum that i managed to achieve was around 74%, usual result something around 64%
  And the best result was achieved with small embedding_vecor_length and small batch_size.</p>
  
  <p>I know - that my test set is only 10 percent of training test, and overall data-set is the biggest problem, but i want to find a way around this problem.</p>
  
  <p>So my questions are -
  1) Is it correctly builded model for text classification purpose? (it works)
  Do i need to use simultaneous convolution an merge results instead?
  I just don't get how the text information doesn't get lost in the process of convolution with different filter sized (like in my example)
  Can you explain hot the convolution works with text data? 
  There are mainly articles about image recognition..</p>
  
  <p>2)i obliviously got a problem with overfitting my model. How can i make the performance better?
  I have already added Dropout layers. What can i do next?</p>
  
  <p>3)May be i need something different? I mean pure RNN without convolution?</p>