Can we run OS X apps inside X11 (XQuartz)?
No. Native OS X apps don't call down to X11 protocol and no one (Apple or other) has implemented any sort of shim/translation layer/conversion tool to port over the API.

It would surely involve slowdown, reduced acceleration, possibly loss of fidelity and loss of functionality.