title: Keras - Text preprocessing
body: <h1>The goal:</h1>
  
  <h2>To generate text from an authors style.</h2>
  
  <p>Input: an authors work to train on, a seed for a prediction</p>
  
  <p>Output: generated text from that seed</p>
  
  <hr>
  
  <h3>Question about the embedding layer in keras:</h3>
  
  <p>I have raw text, a flat text file containing a few thousand lines of text. I want to input this into an embedding layer to keras to vectorize the data. Here is what I have as text:</p>
  
  <pre><code>--SNIP
  The Wild  West\n Ha ha, ride\n All you see is the sun reflectin\' off of the
  --SNIP
  
  and I call it input_text:
  
  num_words = 2000#get 2000 words
  tok = Tokenizer(num_words)#tokenize the words
  tok.fit_on_texts(input_text)#takes in list of text to train on
  #put all words from text into a words array
  #this is essentially enumerating them
  words = []
  for iter in range(num_words):
      words += [key for key,value in tok.word_index.items() if value==iter+1]
  
  #words[:10]
  #Class for vectorizing texts, or/and turning texts into sequences 
  #(=list of word indexes, where the word of rank i in the dataset (starting at 1) has index i).
  X_train = tok.texts_to_sequences(input_text)#turns text to sequence, stating which word comes in what place
  X_train = sequence.pad_sequences(X_train, maxlen=100)#pad sequence, essentially padding it with 0's at the end
  y_train = words
  </code></pre>
  
  <h3>The problem:</h3>
  
  <p>It seems that my code will take in the sequence, then when I apply padding it only gives the first 100 of the sequence. How should I break it apart?</p>
  
  <p>Should I take the entire sequence and go through the first 100 words (X), and give the next one (Y) and do some skips along the way?</p>
  
  <p>I want the output to be the probability of the next word coming up. So I have a softmax layer at the end. Essentially I want to generate text from a seed. Is this the correct way of going about doing this? or is it just better</p>