Mac Mini 2011: Dual Display setup and HDMI Switching
The behavior  you are seeing is because the switch you are using is completely disconnecting the signal from the Mac so macOS assumes (rightly so) that they monitor is gone and it needs to consolidate the windows.

It would be as if you physically removed the cable from the Mac mini and plugged it into your other device.  

To get around this, you need an active device like a KVM (keyboard video mouse switcher) which will maintain the signal(s).  

Though a KVM might be complete overkill if all you really want/need to change video inputs.  What may be the best solution is get a monitor with multiple inputs and you just change it on the monitor itself.