title: Keras word2vec model - average word vectors - best approach
body: <p>I have followed the word2vec tutorial using keras in R <a href="https://blogs.rstudio.com/tensorflow/posts/2017-12-22-word-embeddings-with-keras/" rel="nofollow noreferrer">here</a> and I have managed to obtain word embeddings for each word. I would like to now pass my word embeddings to each <code>document</code> and put the <code>document</code> through some classifier model such as logistic regression. Using the code below I get to this line:</p>
  
  <p><code>row.names(embedding_matrix) &lt;- c("UNK", words$word)</code></p>
  
  <p>I have an embedding matrix which look like the following:</p>
  
  <pre><code>&gt; embedding_matrix[1:5, 1:5]
                    [,1]         [,2]         [,3]         [,4]        [,5]
  UNK         0.03696522  0.040100370  0.005650103 -0.015125941 -0.01914053
  text        0.02520180  0.030634869 -0.003202755 -0.008647407  0.04315760
  to          0.00604762  0.010533611  0.035768602 -0.027628880  0.01015684
  the         0.01279589 -0.048598755  0.044444822  0.046858221  0.01908986
  additional -0.02655490 -0.005862723 -0.002875999 -0.022304043  0.01380521
  </code></pre>
  
  <p>How can I <code>average</code> the word embeddings from this stage?</p>
  
  <p>Code &amp; data generation:</p>
  
  <pre><code># Generate some sample data
  
  text &lt;- c("Because I could not stop for Death I add additional text-",
            "He kindly stopped for me some additional text to act as a filler -",
            "The Carriage held but just Ourselves more additional text to add to the body of the text-",
            "and Immortality plus some more words to fill the text a little")
  
  ID &lt;- c(1,2,3,4)
  output &lt;- c(1,0,0,1)
  df &lt;- data.frame(cbind(ID, text, output))
  df$text &lt;- as.character(df$text)
  
  # Initialise keras parameters
  library(keras)
  tokenizer &lt;- text_tokenizer(num_words = 20)
  tokenizer %&gt;% 
    fit_text_tokenizer(df$text)
  
  
  embedding_size &lt;- 128  # Dimension of the embedding vector.
  skip_window &lt;- 5       # How many words to consider left and right.
  num_sampled &lt;- 1       # Number of negative examples to sample for each word.
  
  input_target &lt;- layer_input(shape = 1)
  input_context &lt;- layer_input(shape = 1)
  
  embedding &lt;- layer_embedding(
    input_dim = tokenizer$num_words + 1, 
    output_dim = embedding_size, 
    input_length = 1, 
    name = "embedding"
  )
  
  target_vector &lt;- input_target %&gt;% 
    embedding() %&gt;% 
    layer_flatten()
  
  context_vector &lt;- input_context %&gt;%
    embedding() %&gt;%
    layer_flatten()
  
  dot_product &lt;- layer_dot(list(target_vector, context_vector), axes = 1)
  output &lt;- layer_dense(dot_product, units = 1, activation = "sigmoid")
  
  model &lt;- keras_model(list(input_target, input_context), output)
  model %&gt;% compile(loss = "binary_crossentropy", optimizer = "adam")
  
  summary(model)
  
  
  
  ######################## Broken skigrams_generator, have to define our own #######################
  
  skipgrams_generator &lt;- function(text, tokenizer, window_size, negative_samples) {
    gen &lt;- texts_to_sequences_generator(tokenizer, sample(text))
    function() {
  
      while(TRUE) {
        nxt &lt;- generator_next(gen)
        if (length(nxt) &gt; 1)
          break
      }
  
      skip &lt;- nxt %&gt;%
        skipgrams(
          vocabulary_size = tokenizer$num_words, 
          window_size = window_size, 
          negative_samples = 1
        )
      x &lt;- transpose(skip$couples) %&gt;% map(. %&gt;% unlist %&gt;% as.matrix(ncol = 1))
      y &lt;- skip$labels %&gt;% as.matrix(ncol = 1)
  
      list(x, y)
    }
  }
  
  ########################################################
  
  # Run the word2vec model
  model %&gt;%
    fit_generator(
      skipgrams_generator(df$text, tokenizer, skip_window, negative_samples), 
      steps_per_epoch = 2, epochs = 2
    )
  
  library(dplyr)
  
  # Get the word embeddings
  embedding_matrix &lt;- get_weights(model)[[1]]
  
  words &lt;- data_frame(
    word = names(tokenizer$word_index), 
    id = as.integer(unlist(tokenizer$word_index))
  )
  
  words &lt;- words %&gt;%
    filter(id &lt;= tokenizer$num_words) %&gt;%
    arrange(id)
  
  row.names(embedding_matrix) &lt;- c("UNK", words$word)
  </code></pre>
  
  <p>I have followed a different tutorial which uses the Word2Vec model from the <code>h2o</code> package. The <code>h2o</code> packages allows me to <code>average</code> the documents word embeddings using <code>h2o.transform(w2v.model, words, aggregate_method = "AVERAGE")</code>. I would like to know what is the best approach for me to do this once I get the word embeddings from the <code>keras</code> model, as I am a little unfamiliar with the <code>h2o</code> package and <code>h2o environments</code></p>
  
  <pre><code>text &lt;- c("Because I could not stop for Death I add additional text-",
            "He kindly stopped for me some additional text to act as a filler -",
            "The Carriage held but just Ourselves more additional text to add to the body of the text-",
            "and Immortality plus some more words to fill the text a little")
  
  ID &lt;- c(1,2,3,4)
  output &lt;- c(1,0,0,1)
  df &lt;- data.frame(cbind(ID, text, output))
  df$text &lt;- as.character(df$text)
  
  library(h2o)
  
  h2o.init()
  df &lt;- as.h2o(df)
  words &lt;- h2o.tokenize(df$text, "\\\\W+")  # Tokenize the words in the text
  # apply the h2o word to vec model
  w2v.model &lt;- h2o.word2vec(words,
                            model_id = "w2v_model",
                            vec_size = 100,
                            word_model = c("SkipGram"),
                            min_word_freq = 1,
                            window_size = 5,
                            init_learning_rate = 0.025,
                            sent_sample_rate = 0,
                            epochs = 1)
  # Avergae the word ebeddings using the h2o package
  averaged_docs &lt;- h2o.transform(w2v.model, words, aggregate_method = "AVERAGE")
  # join the Y variable and X variables together
  ML_data &lt;- h2o.cbind(as.factor(df["output"]), averaged_docs)
  
  averaged_docs &lt;- as.data.frame(averaged_docs)
  
  dim(ML_data)
  # NOTE here the code doesn't work since I give the model too few observations (it does work on larger datasets, I include it
  # here for completness)
  gbm.model &lt;- h2o.gbm(x = names(averaged_docs), y = "output",
                       training_frame = ML_data[1:101], validation_frame = NULL)
  </code></pre>