How is the word embedding represented in the paper "Recurrent neural network based language model"?
Input vector contains two concatenated parts. The low part represents the current word:


  word in time t encoded using 1-of-N coding [...] - size of vector x is
  equal to size of vocabulary V (this can be in practice 30 000-200 000) plus [...]


where, as you said, 1-of-N means (see here, 1-of-V):


  If you have a fixed-size vocabulary of symbols with V members in
  total, each input symbol can be coded as a vector of size V with all
  zeros except for the element corresponding to the symbol's order in
  the vocabulary, which gets a 1.


The high part of the input vector represents the current context:


  and previous context layer [...] Size of context (hidden) layer s is
  usually 30 - 500 hidden units.
  
  For initialization, s(0) can be set to vector of small values, like
  0.1


the article includes this expression:


  x(t) = w(t) + s(t - 1)


that I think better to write as:


  x(t) = w(t) || s(t - 1)


to made more visible the concatenation.

Finally, paper states some improvements that breaks the 1-to-N definition of the word (low) part, in order to reduce the size of the w vector:


  we merge all words that occur less often than a threshold (in the
  training text) into a special rare token.