How to write out the definition of the value function for continous action and state space
this can be written as the integral, is this correct?


Yes. Your derivations imply that we have assumed a deterministic reward given current state-action $(\boldsymbol{s},\boldsymbol{a})$. An stochastic reward model would be $p(\boldsymbol{s}', r|\boldsymbol{s},\boldsymbol{a})$ which requires an additional integral over reward $r$ (for example, equation (3.14) page 47)


  Are my integral versions correct?


Yes. You are unfolding the recursive definition. An illustration would be the recursive definition for factorial:
$$f(n) = nf(n-1);f(0)=1$$
Which is unfolded as:
$$f(n) = n [(n-1) [(n-2)[...]]]$$
However, the difference is that the index in Bellman equation  is going forward since current value depends on future values not the previous ones.