Lego 3804, Mindstorms Robotics Invention System 2.0 on Windows 10
That set has an RCX in it, is that right?  I have several suggestions for you:


Have you tried running the software/installer file in compatibility mode?  I understand this software was designed for Windows 95 or 98.  If this is the case, there is an option in Windows to 'run in compatibility mode' by right clicking the .exe file, selecting 'properties' and then under the compatibility section, select whichever operating system the software was originally designed for. 
I have found this article which may provide you with a
tutorial.  I am not personally familiar with the set, but I assume this software will work.
Failing all else, I suggest you try Brixc Command Centre .  This uses a derivation of C for programming your robot which, as I understand is not too different from the original Lego software.  This is freeware and allows you to program not only the RCX, but other versions of Mindstorms robots if that is of interest.  I personally use this all the time with my Mindstorms NXT sets and it works a treat.  It's a great way to introduce programming to people as well!


Hope these suggestions are of help!