Are there any modern X11/Xorg Browsers that compile for OSX?
I'm not sure your reason for wanting a browser under X11, but your needs might be met by running Chromium or Firefox in a Docker container over VNC. Then, you can use any window manager you want.

For example, here's my container that exposes Firefox through OpenBox. 

Firefox over VNC on Docker Hub