The task of predicting the next token in a sequence can be framed as a partially observable Markov Decision Process, lending itself to the use of RL as an approach to learn which decisions to make in the process to maximize an arbitrary reward. Where each token 𝑥 𝑖 is considered a state out of all possible states 𝑠 ∈ 𝑆 at a given timestep in a trajectory 𝑡 ∈ 𝜏, the transition between two states is determined by an action out of all possible actions 𝑎 ∈ 𝐴. Combining this framework with our generative model, the pre-trained network that learns the probability 𝑃(𝑥 𝑖 |𝑥 𝑖-1 ,… , 𝑥 0 ) can equally be considered an initialized policy where the policy function parameterized by the network weights denotes the probability of taking an action given the state at a given timestep 𝜋 𝜃 (𝑎 𝑡 |𝑠 𝑡 ). Within this framework, many different RL algorithms exist to modify the policy such as to optimize an arbitrary reward 𝑅.