How to run remote applications on OS X like X11 does?
Short answer is no. 

Cocoa is not designed as a remote procedure call (RPC)-esque protocol, while X is. You cannot tunnel the interfaces of Cocoa applications like you know it from X11 natively, only using protocols like VNC that transfer the actual bitmap rather than the rendering instructions.

On the other hand, you are able to use remotely running X applications via SSH.