What is the most biologically plausible representation for the actor and critic?
For the actor, I'd say the 'not mutually exclusive' option is more biologically plausible in the context of muscle systems, where the actions can be seen as simultaneous muscle activations. Maybe at a higher level, an agent thinks of the action as 'go north' or 'go south', but the final outputs which have to control muscles at a lower level have to represent simultaneous muscle activations.

For the critic, I'd say the 'single output node' is more biologically plausible. Agents perceive the world in the form of high dimensional inputs, such as images. The approach where a value function is learned in a tabular fashion where you know the value for every single state doesn't really scale very well and is limited to small discrete state spaces. For biological agents, it makes sense to have a function that senses the current state of the environment and outputs a single number that represents the value, which gives the agent an idea of how things are going so far given the actions it took in the past.