title: Why does embedding vector multiplied by a constant in Transformer model?
body: <p>I am learning to apply <strong>Transform model</strong> proposed by <a href="https://arxiv.org/abs/1706.03762" rel="noreferrer">Attention Is All You Need</a> from tensorflow  official document <a href="https://www.tensorflow.org/beta/tutorials/text/transformer" rel="noreferrer">Transformer model for language understanding</a>.</p>
  
  <p>As section <a href="https://www.tensorflow.org/beta/tutorials/text/transformer#positional_encoding" rel="noreferrer">Positional encoding</a> says:</p>
  
  <blockquote>
    <p>Since this model doesn't contain any recurrence or convolution,
    positional encoding is added to give the model some information about
    the relative position of the words in the sentence.</p>
    
    <p><strong>The positional encoding vector is added to the embedding vector</strong>.</p>
  </blockquote>
  
  <p>My understanding is to add <code>positional encoding vector</code> directly to <code>embedding vector</code>. But I found <code>embedding vector</code> multiplied by a constant when I looked at the code.</p>
  
  <p>The code in section <a href="https://www.tensorflow.org/beta/tutorials/text/transformer#encoder" rel="noreferrer">Encoder</a> as follows:</p>
  
  <pre><code>class Encoder(tf.keras.layers.Layer):
    def __init__(self, num_layers, d_model, num_heads, dff, input_vocab_size, 
                 rate=0.1):
      super(Encoder, self).__init__()
  
      self.d_model = d_model
      self.num_layers = num_layers
  
      self.embedding = tf.keras.layers.Embedding(input_vocab_size, d_model)
      self.pos_encoding = positional_encoding(input_vocab_size, self.d_model)
  
  
      self.enc_layers = [EncoderLayer(d_model, num_heads, dff, rate) 
                         for _ in range(num_layers)]
  
      self.dropout = tf.keras.layers.Dropout(rate)
  
    def call(self, x, training, mask):
  
      seq_len = tf.shape(x)[1]
  
      # adding embedding and position encoding.
      x = self.embedding(x)  # (batch_size, input_seq_len, d_model)
      x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))
      x += self.pos_encoding[:, :seq_len, :]
  
      x = self.dropout(x, training=training)
  
      for i in range(self.num_layers):
        x = self.enc_layers[i](x, training, mask)
  
      return x  # (batch_size, input_seq_len, d_model)
  </code></pre>
  
  <p>We can see <code>x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))</code> before <code>x += self.pos_encoding[:, :seq_len, :]</code>.</p>
  
  <p>So why does embedding vector multiplied by a constant before adding positional encoding in Transformer model?</p>