Is it possible to connect Nexus 4 to Windows Server 2012 R2?
Actually Windows Experience Feature is the answer!

I had the same problem and could not install usb drivers for a MotoG on a win2012 Server, but when I installed Desktop Experience feature, all runs flawlessly, drivers auto-installed and all!

BUT it is a little weird to find it...

Go to:

Server Manager -> Open Add Roles and Features Wizard -> Features -> User Interfaces and Infrastructure -> Desktop Experience