title: Understanding `tf.nn.nce_loss()` in tensorflow
body: <p>I am trying to understand the NCE loss function in Tensorflow. NCE loss is employed for a word2vec task, for instance:</p>
  
  <pre><code># Look up embeddings for inputs.
  embeddings = tf.Variable(
      tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0))
  embed = tf.nn.embedding_lookup(embeddings, train_inputs)
  
  # Construct the variables for the NCE loss
  nce_weights = tf.Variable(
      tf.truncated_normal([vocabulary_size, embedding_size],
                          stddev=1.0 / math.sqrt(embedding_size)))
  nce_biases = tf.Variable(tf.zeros([vocabulary_size]))
  
  # Compute the average NCE loss for the batch.
  # tf.nce_loss automatically draws a new sample of the negative labels each
  # time we evaluate the loss.
  loss = tf.reduce_mean(
      tf.nn.nce_loss(weights=nce_weights,
                     biases=nce_biases,
                     labels=train_labels,
                     inputs=embed,
                     num_sampled=num_sampled,
                     num_classes=vocabulary_size))
  </code></pre>
  
  <p>more details, please reference Tensorflow <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/word2vec/word2vec_basic.py" rel="noreferrer">word2vec_basic.py</a></p>
  
  <ol>
  <li>What are the input and output matrices in the NCE function? </li>
  </ol>
  
  <p>In a word2vec model, we are interested in building representations for words. In the training process, given a slid window, every word will have two embeddings: 1) when the word is a centre word; 2) when the word is a context word. These two embeddings are called input and output vectors, respectively. (<a href="http://www-personal.umich.edu/~ronxin/pdf/w2vexp.pdf" rel="noreferrer">more explanations of input and output matrices</a>)</p>
  
  <p>In my opinion, the input matrix is <code>embeddings</code> and the output matrix is <code>nce_weights</code>. Is it right?</p>
  
  <ol start="2">
  <li>What is the final embedding?</li>
  </ol>
  
  <p>According to a <a href="https://stackoverflow.com/questions/37982478/what-is-the-purpose-of-weights-and-biases-in-tensorflow-word2vec-example">post</a> by s0urcer also relating to <code>nce</code>, it says the final embedding matrix is just the input matrix. While, <a href="https://stats.stackexchange.com/questions/177667/input-vector-representation-vs-output-vector-representation-in-word2vec">some others saying</a>, the <code>final_embedding=input_matrix+output_matrix</code>. Which is right/more common?</p>