How do I begin to approach this dynamic discrete choice model?
$b_t$ is the decision variable, and the value function is formulated as a function of the state variable of the problem. In your case, the role of the state variable is played by the $p$ parameter, which may change in each period (updated). So your Bellman equation is

$$V(p_t) = \max_{b_t} \{u(b_t) + \beta E [V(p_{t+1})\mid t]\}$$

and I guess now you see where the updating of $p$ enters the picture.

On "Value function iteration", or "iteration on the Bellman equation", the book "Recursive Macroeconomic Theory" of Ljungqvist &amp; Sargent contains clear and practical guidance.