IC buffer like 4066 with 8 or 16 switches
Here's a collection of various 15-pin VGA schemes: -



From what I can tell, these are the following lines that probably need switching: -


Red video (pin1)
Green Video (pin 2)
Blue video (pin 3)
Horizontal sync (pin 13)
Vertical sync (pin 14)


On some devices pins 4, 11 and 12 are no-connects i.e. they do nothing (or are reserved) and 5, 6, 7, 8 and 10 are ground. Pin 9 can be +5V and pin 11 is sometimes "monitor ID"

On other devices pins 12 and 15 look like they perform some kind of SPI serial function.

I would try and make a switch that accomodates 1, 2, 3, 13 and 15 as change-overs but I reckon using a 4066 may not be fast enough - look for a video switch element.

Important PSU info
It is possible that you will need an analogue switch that is powered above ground (5V typically) and below ground (-5V) to accodate the AC nature of the video signals.

As for finding a device, this may be suitable as it is a 3-port switch with 7 channels: -



The blurb says it is "designed for 7-channel VGA signals (R, G, B Hsync, Vsync, DDC Data and DDC clock). By virtue of it operating only from 3V3 I have to assume the RGB signals are biased above GND. This is something you need to determine for sure but it looks promising.