How to prevent installing module in already installed module?
How about this?


Create a central module that implements hook_module_preinstall
Follow some naming convention for the modules which should not coexist, or simply gather their names in a list
Use your naming convention or the list of module names in your hook to detect installation of one of these modules. Also, implement the logic for counting the number of such modules that are already enabled in the system.
Conditional on one such module arriving as an argument to hook_module_preinstall and the count of existing enabled modules of this kind being greater than 0, throw a custom Exception in order to block the installation of the said module.
Most importantly, add your central module as a dependency to all of your modules that you wanna control in that way.


Very theoretical answer. :-) 
May it be of help to you!