What is the positional encoding in the transformer model?
For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be
$$\begin{align*}e_{w}' &amp;= e_{w} + \left[sin\left(\frac{pos}{10000^{0}}\right), cos\left(\frac{pos}{10000^{0}}\right),sin\left(\frac{pos}{10000^{2/4}}\right),cos\left(\frac{pos}{10000^{2/4}}\right)\right]\\
&amp;=e_{w} + \left[sin\left(pos\right), cos\left(pos\right),sin\left(\frac{pos}{100}\right),cos\left(\frac{pos}{100}\right)\right]\\
\end{align*}$$
where the formula for positional encoding is as follows
$$\text{PE}(pos,2i)=sin\left(\frac{pos}{10000^{2i/d_{model}}}\right),$$
$$\text{PE}(pos,2i+1)=cos\left(\frac{pos}{10000^{2i/d_{model}}}\right).$$
with $d_{model}=512$ (thus $i \in [0, 255]$) in the original paper.
This technique is used because there is no notion of word order (1st word, 2nd word, ..) in the proposed architecture. All words of input sequence are fed to the network with no special order or position; in contrast, in RNN architecture, $n$-th word is fed at step $n$, and in ConvNet, it is fed to specific input indices. Therefore, proposed model has no idea how the words are ordered. Consequently, a position-dependent signal is added to each word-embedding to help the model incorporate the order of words. Based on experiments, this addition not only avoids destroying the embedding information but also adds the vital position information.
This blog by Kazemnejad explains that the specific choice of ($sin$, $cos$) pair helps the model in learning patterns that rely on relative positions. As an example, consider a pattern like
if 'are' comes after 'they', then 'playing' is more likely than 'play'

which relies on relative position &quot;$pos(\text{are}) - pos(\text{they})$&quot; being 1, independent of absolute positions $pos(\text{are})$ and $pos(\text{they})$. To learn this pattern, any positional encoding should make it easy for the model to arrive at an encoding for &quot;they are&quot; that (a) is different from &quot;are they&quot; (considers relative position), and (b) is independent of where &quot;they are&quot; occurs in a given sequence (ignores absolute positions), which is what $\text{PE}$ manages to achieve.
This article by Jay Alammar explains the paper with excellent visualizations. The example on positional encoding calculates $\text{PE}(.)$ the same, with the only difference that it puts $sin$ in the first half of embedding dimensions (as opposed to even indices) and $cos$ in the second half (as opposed to odd indices). As pointed out by ShaohuaLi, this difference does not matter since vector operations would be invariant to the permutation of dimensions.