To clarify the discussion to specifically address actor-critic policy-gradient reinforcement learning methods, it is important to note that not all policy-gradient methods employ a critic. It is also recommended to use precise language when describing the agent's view of the environment by using the term "observation" rather than "state," to avoid assuming that the environment is a fully observable Markov Decision Process.

In the context of actor-critic policy-gradient RL, the algorithm learns two components: an actor function and a critic function. For problems with non-trivial action and observation spaces, deep neural networks are commonly used to represent the actor and critic functions. Specifically, the actor network maps each possible observation in the agent's observation space to a probability distribution over possible actions in the action space. Meanwhile, the critic network maps each possible observation to an expected value estimate.

In practice, some of the initial layers are often shared between the actor and critic networks. This is because the networks need to perform similar processing tasks, such as edge detection for pixel-valued observation spaces. However, this sharing of layers is not strictly required, as the networks are trained to produce different outputs.
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence