How to create a statemachine/wizard in a modal/popup?
I'm not saying that civi statemachine has been done a decade ago and that it shows, but most of us would rather do something else than trying to use it ;)

How familiar are you with angular? You would end up with a nicer and smoother interface, and angular is now part of the core and with lots of useful helpers to interface with the api.

Look what @totten has done for the new civimail interface on 4.6 for instance for inspiration and an example of modal interface.

He and robin did a session about angular at civicon london that you can watch online:
https://london2014.civicrm.org/sessions/using-angularjs-enhance-user-experience