title: Keras LSTM Language Model using Embeddings
body: <p>I am doing a language model using keras. </p>
  
  <p>Basically, my vocabulary size N is ~30.000, I already trained a word2vec on it, so I use the embeddings, followed by LSTM, and then I predict the next word with a fully connected layer followed by softmax. 
  My model is written as below : </p>
  
  <pre><code>EMBEDDING_DIM = 256
  embedding_layer = Embedding(N,EMBEDDING_DIM,weights=[embeddings],
  trainable=False)
  
  model = Sequential()
  model.add(embedding_layer)
  model.add(LSTM(EMBEDDING_DIM))
  model.add(Dense(N))   
  model.add(Activation('softmax')) 
  
  model.compile(loss="categorical_crossentropy", optimizer="rmsprop")
  </code></pre>
  
  <p>I have two questions :</p>
  
  <ol>
  <li><p>in this case, can you confirm that we only use the last hidden layer of the LSTM (which is followed by the fully connected layer and softmax) and there isn't something like a max/mean-pooling of successive hidden layers of the lstm (like here for sentiment analysis <a href="http://deeplearning.net/tutorial/lstm.html" rel="noreferrer">http://deeplearning.net/tutorial/lstm.html</a>) ?</p></li>
  <li><p>What do you think, instead of connecting the last hidden layer of the lstm to a big fully connected layer of size N (30.000), connecting to a layer of size EMBEDDING_DIM, and predicting the embedding of the next word instead of the word itself, in which case we replace the loss by something like mse, reducing training time, and mainly "helping" our model because the vocabulary is big and embeddings can be useful also for the end of the network ?</p></li>
  </ol>
  
  <p>Thanks !  </p>