What is the difference between the state transition of an MDP and an action-value?
Transition Probabilities: Consider that you are at state $s$ and from that state take an action $a$.Then there are some probability you will land up at state $s_{1}'$ or $s_{2}'$ ($s'$ indicate the next states). Those probabilities are called transition probabilities. In this example, the transition matrix is just a 3D array since it depends on your state and action($p(s, a)$).

Action value function $Q_{\pi}(s, a)$: It is the expected total reward you get from state $s$, taking action $a$ and thereafter following the policy, $\pi$. 


  Is the state transition in MDP stochastic transition, meaning transition to some other state without taking any action?


The environment can be stochastic or deterministic. If the environment is stochastic then those transitions are stochastic. If the environment is deterministic then those transitions are deterministic.