Commerce checkout does not work when loading from AJAX
I had exactly the same Problem, i think this page should help you: Loading a form by ajax.

I have solved this problem, but I think its a little bit ugly. The Problem was that I have loaded the checkout form, but not the scripts which are necessary on the checkout page. So i have loaded the scripts from the checkout page, too. After that i have done a 

Drupal.attachBehaviors();


So all scripts can attach there functionality to the checkout form.