When am I allowed to drag a space/app between displays?
After playing around with it some more, I have discovered the pattern.  (It was tricky to figure out because it changes!)

You can't drag the current desktop (of any display) off of that display.