Should I install X11 on my Mac Book Pro?
If you install the X11 subsystem, it will only run when you have an X11 program running.

Much the same as when installing the JVM - that will only run when you run a Java program.