Influenced by recent advances in natural language processing (NLP), autoregressive language models based on recurrent neural networks or transformer architectures are commonly employed for this purpose. Given the preceding tokens x <t = (x 1 , x 2 , . . . , x t-1 ) in a sequence, these models compute the logits for the t-th token, denoted as z t ∈ R |V| , where V represents the vocabulary of tokens. The logits are then normalized using the softmax function to obtain the probability distribution over the vocabulary: