In Transformer machine learning models, attention is a mechanism used to weigh the importance of different input elements when processing sequences. In standard attention, a single set of learned weights is used to compute a weighted sum of the input elements, where the weights are determined by the similarity between the input element and a query vector.

Multi-head attention extends the standard attention mechanism by performing multiple parallel attention operations, or "heads". Each head learns a separate set of weights and attends to a different representation of the input sequence. The outputs of each head are then concatenated and transformed to produce the final output.

From a mathematical perspective, in multi-head attention, the query, key, and value vectors are each split into multiple smaller vectors, or "heads", which are then processed in parallel. The output of each head is computed using a scaled dot-product attention mechanism, where the weights are determined by the similarity between the query and key vectors for that head.

The outputs of each head are then concatenated and passed through a linear layer to produce the final output. This allows the model to capture different types of relationships between the input elements and the query vector, improving its ability to represent complex input sequences.
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence