How to run Dockerized graphical applications on a headless server with packages required for display remotely on clients as part of a separate image?
You can't run a program running on x11 without a x11 buffer somewhere, be it vnc, xvfb or whatever. You will always need x11 libraries on the system running the program or it couldn't work at all. That's akin to  trying to run a C program without libc on the system. 

Using system libraries from a different container with a volume is a bad idea, as it create a hard dependency on both containers using the same libraries, and is likely to crash somewhere when both systems try to acquire a lock on the library to load it.

I assume with a lot of tweaks it should be doable, but the added complexity is a real footgun and create more dependencies than should exist.