Permanently set Startup Disk to Windows without Boot Camp utility?
You should not of had to install the drivers manually. If you right click on setup.exe from the "Boot Camp Support Software", you should be able to select compatibility mode. You say you formatted the drive. It would help if you explained what utility you used to do this. My concern is for how Windows determines the drive partitioning. For your model/year Mac, Windows should be installed in MBR/BIOS mode. 

If the "Boot Camp Support Software" is installed correctly, then you can select the startup disk from the Boot Camp pane on the Control Panel.

Another option which works on most later Macs, is to hold down the alt/option key on startup. When the icon labeled Windows appears, hold down the control key and select Windows. This will make Windows the default operation system.

Here Is Some Advice on the Boot Camp Support Software.

According to the Apple web site "Install Windows 7 and earlier on your Mac using Boot Camp", the proper "Boot Camp Support Software" (BCSS) for installing 64 bit Windows 7 on your model Mac is "Boot Camp Support Software 5.1.5621". According to the Apple web site "Use Windows 8.1 on your Mac with Boot Camp", your model also supports 64 bit Windows 8.1. For this version of Windows, Apple requires you to use the Boot Camp Assistant to download the BCSS to a flash drive. You can download this software without installing Windows 8.1 to your Mac.

One of the machines, I have access to, is a 2011 iMac. It uses the same version of BCSS as your Mac mini for the installation of 64 bit Windows 7. I verified that the BCSS for installing 64 bit Windows 8.1 on this iMac is exactly the same as for installing 64 bit Windows 7. I have no way of verifying if is this is also true for your Mac mini.


  I thought I used the procedure given in the comments below to run the BCSS on the 2011 iMac with 64 bit Windows 10 installed. Now that I think about it, I may not of had to.


Eventually, the BCSS is released in sets of versions. As far as I can tell, only version sets 4, 5 and 6 can be downloaded from an Apple web site and/or through the Boot Camp Assistant. Older sets exist on the Snow Leopard (OS X 10.6) or older DVDs. Some (if not all) of the version 4 BCSS downloads contain a Bootcamp64.msi file. I know this to be true for 4.0.4033 and 4.0.4255. Most (if not all) of the version 5 BCSS downloads do not contain a Bootcamp64.msi file. I know this to be true for 5.1.5621 and 5.1.5640. The version 6 set is irrelevant, since Apple does not officially support Windows 10 on 2011 Macs.


  This leads me to the conclusion that you are using the wrong BCSS.


You should probably install El Capitan (OS X 10.11) and download the BCSS using the Boot Camp Assistant. Or at least, you should be download "Boot Camp Support Software 5.1.5621".