Why doesn't stability in prediction imply stability in control in off-policy reinforcement learning?
It is a mathematical problem. Usually, we use contraction mapping theorem for the proof of convergence. You should apply the Banach fixed point theorem for Bellman's functions.