Is there any way to not boot Finder at startup?
I don't know if this still works; but in 10.6 you could define a different application to run in place of Finder.  I did this for a iChat stand-alone "kiosk" once.

For example:

defaults write com.apple.loginwindow Finder /Applications/iChat.app


When you logout and login, the application you designated (in my case, iChat) will start.  I suppose you could set this to XQuartz; but, again, I haven't tried it myself.

To back out the changes:

defaults delete com.apple.loginwindow Finder