How many DBMS's can be installed on a single server?
Separate.

SQL Server, Oracle, DB2 et al typically do not play nicely alongside other "server" software. They tend to monopolise system resources, particularly memory, so while you could install them side-by-side, you won't be getting the best from them. It would also be challenging to diagnose performance issues with two different DBMS's competing for resources.

Given a collection of ageing or little used applications that required a mix of database systems, virtualisation would be a tempting solution to consolidate.