How to open User registration in a pop up?
Probably your server's settings prevent opening registration pages within frames. This prevents embedding your registration form in 3rd party webpage to create spam users et cetera. That is usually a good thing, even if what happens to you is over-restrictive.

Now, why to reinvent the wheel? There is a Modal forms module ready.


  Modal forms make use of the modal feature in the ctools module to open some common forms in a modal window.
  Supported forms
  
  
  Log in (modal_forms/nojs/login)
  Request new password (modal_forms/nojs/password)
  Create new account (modal_forms/nojs/register)
  Contact (modal_forms/nojs/contact)
  Comment (modal_forms/nojs/comment/reply/%node)
  Webform (modal_forms/nojs/webform/%node)
  


Emphasis mine.

It has some bugs, but for first 3 it works pretty well. Or at least it used to.