Ctools nested modals
I think you need to create your own ajax handler for this. The idea would be to check if an existing modal exists and rather than add a new one, simply run a replace in your existing one. If non exists, you'd need to setup a new modal.

The other approach you can take, is rather than just forcing it to opena new one, send an ajax request to close the existing one and then present the new one.

Either way, I do not believe there is an easy or straight solution for this particular problem.