Not all policy-gradient methods use a critic, so I changed the language to specify you are talking about actor-critic policy-gradient RL.  I also added some context about how there are often shared layers between the actor and critic networks. A minor nitpick is that it's good to be precise about whether you are using the term 'state' or the term 'observation' to describe the agent's view of the environment. I chose to use the term 'observation' because I think it's most precise to not assume that the environment is a fully observable Markov Decision Process. 

With this in mind, here is my rewrite:


>When employing a actor-critic policy-gradient reinforcement learning method, the algorithm learns two components, an actor and a critic function. For problems with non-trivial action and observation spaces, the actor and critic functions are usually represented by deep neural networks. The actor network maps each possible observation in the agent's observation space to a probability distribution over possible actions in the action space. The critic network maps each possible observation to a corresponding expected value estimate. In practice, some of the early layers are often shared between the actor and critic networks, though this is not required.  This is based on the intuition that some of the same processing will need to be done by both the actor and the critic network, though they are trained to produce different outputs. An example of this is that if working with a pixel-valued observation space, both the actor and critic networks will need to learn 
edge-detection.
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence