Where d k is the keys dimension, T is the transpose operation and the softmax function serves as a scoring function to map the logits (QK T ) into probabilities that serve as the attention values. Multiplying by the values (V) serves to produce the scaled embedding values. Graphormer utilizes a multi-headed attention mechanism akin to the original transformer architecture, where h attention heads calculate parallelized attention scores. First, input data is decomposed into the query, keys, and values which consist of h sets of learned linear projections. Attention scores are then calculated in parallel for each head and then concatenated and projected back into embedding space to produce the final unified score (shown in the centrality encoding in Figure ). Multi-headed attention is thought to improve on dot product attention because of the model's ability to simultaneously understand different types of relationships in the input that govern relative importance. Talking Heads (TH) Attention.