title: NLP - Open Vocabulary Word Embedding
body: <p>How can I convert words to vectors (Word embedding) if I don't have a predefined dictionary of words? Most word embedding implementations like <a href="https://code.google.com/p/word2vec/" rel="nofollow">Word2vec</a> and <a href="http://nlp.stanford.edu/projects/glove/" rel="nofollow">GloVe</a> have a fixed dictionary of words. The input to the neural networks are one-hot encoded and the hidden layer sizes also depend on the vocab size, which makes adding a new word later on without re-training all the vectors again impossible. I need a network that outputs a fixed dimensional vector for any arbitrary word input. But how to input the 'word' into the network? one hot encoding is not possible, as i don't have a fixed dictionary of words.</p>
  
  <p>Will converting the word to a trigram vector or a bigram vector work? Trigram vectors have been used for Sentence embedding (<a href="http://arxiv.org/pdf/1502.06922.pdf" rel="nofollow">Deep Sentence Embedding Using Long Short-Term Memory Networks</a>), but I doubt if it will work equally as well for word embedding, as there are changes in both the network architecture (Word embedding uses shallow network whereas sentence embedding uses RNNs) and the auxiliary task. Please help.</p>
  
  <p>Note:</p>
  
  <p>By "converting to trigram vector" I mean the following :</p>
  
  <ul>
  <li>Let the input word be "CAT" Add #s at the beginning and at the end :
  "#CAT#"</li>
  <li>List all the possible tri-grams:  #CA, CAT, AT#</li>
  <li>Each trigram is converted to a one hot encoded vector of dimension NxNxN 
  where N is my character set size. eg., E("#CA") = {0,0,0,0,0,...,0,<strong>1</strong>,0,0,0} </li>
  <li>The one hot encoded vector of every trigram of the word is added to
  get the "tri-gram vector" of the word.
  e.g., trigram_vec("CAT") = {0,0,0,0,...0,<strong>1</strong>,0,0,...0,0,<strong>1</strong>,0,...0,0,<strong>1</strong>,0,0,0,0}</li>
  </ul>
  
  <p>Thanks for any help in advance!</p>