In reinforcement learning, does the optimal value correspond to performing the best action in a given state? I am confused about the definition of the optimal value ($V^*$) and optimal action-value (Q*) in reinforcement learning, so I need some clarification, because some blogs I read on Medium and GitHub are inconsistent with the literature.
Originally, I thought the optimal action value, $Q^*$, represents you performing the action that maximizes your current reward, and then acting optimally thereafter.
And the optimal value, $V^*$, being the average $Q$ values in that state.  Meaning that if you're in this state, the average &quot;goodness&quot; is this.
For example:
If I am in a toy store and I can buy a pencil, yo-yo, or Lego.
Q(toy store, pencil) = -10
Q(toy store, yo-yo) = 5
Q(toy store, Lego) = 50

And therefore my $Q^* = 50$
But my $V^*$ in this case is:
V* = -10 + 5 + 50 / 3 = 15

Representing no matter what action I take, the average future projected reward is $15$.
And for the advantage of learning, my baseline would be $15$.  So anything less than $0$ is worse than average and anything above $0$ is better than average.
However, now I am reading about how $V^*$ actually assumes the optimal action in a given state, meaning $V^*$ would be 50 in the above case.
I am wondering which definition is correct.
I am wondering which definition is correct.

The asterisk * in both the definitions stands for &quot;optimal&quot; in the sense of &quot;value when following the optimal policy&quot;
So this one is correct:

$V^*$ actually assumes the optimal action in a given state, meaning $V^*$ would be $50$ in the above case

However, you have got the definition of Q slightly wrong.
I think this is because you are omitting the parameters.
The state value function uses the state as a parameter, $V_{\pi}(s)$, it returns the value of being in state $s$ and following a fixed policy $\pi$. The * is used to denote following an optimal policy.
The action value function has two parameters - a state and an action that is possible in that state, $Q_{\pi}(s, a)$, it returns the value of being in state $s$, taking action $a$ (regardless of whether it is the best action or not) and following the policy $\pi$ after that point.
Your assertion in the question:

And therefore my $Q^* = 50$

is wrong, or rather not meaningful, as you have not stated the parameters. You already list all the possible values of Q with the parameters. You could say $\text{max}_a Q(\text{toy store}, a) = 50$, or to choose the best action $\pi(\text{toy store}) = \text{argmax}_a Q(\text{toy store}, a) 
= \text{Lego}$