The relative importance of each input is determined by the score function, which compares the current hidden state of the LSTM with each input in the sequence. The score function can take different forms, but commonly used ones include dot product, additive attention, and multiplicative attention. The score function produces a score for each input, indicating its compatibility with the current hidden state. These scores are then normalized to produce a probability distribution over the inputs, using a softmax function. This probability distribution represents the relative importance of each input to the output, with higher probabilities indicating greater importance. The weighted sum of the inputs is then computed using this probability distribution, giving more weight to the inputs that are more important for the current output.
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence