Conflict between ajax callback and jQuery UI dialog
a simpler approach might be to include the form in a custom block, and set it's css to match the modal design you want but set the initial block display to none.  Then use an onclick to show the block using whatever transition you want