Is there an official Android "Google" ROM? So I have an old unlocked phone and I'd like to load up my own OS.  I've seen some mods out there.  CyanogenMod seems to be the most popular.  However, I was curious if I could just install a pure untouched version of the OS -- as distributed by Google. I seem to remember being able to download a clean compiled version of the Android OS.  

Does Google distribute any binaries like this, and if not, is there someone out there that does?  

Additionally, is this even worth trying?  These days is it necessary for Android to be packaged with special drivers and such for all of the different hardware out there?
To answer if Google distributes binaries for your device: No, they do not (Unless you have a Nexus device). They do, however, have the full Android source code available so that if you wanted you could compile the source code for your device. However, that leads into your second question about the drivers and such. Yes, you will need to find the drivers and any other proprietary files that your device requires for it to interface with Android. You would have to search about compiling a custom ROM for your device and how to include these files. 

Likely, your simplest choice is find an AOSP (Android Open Source Project) ROM that exists for your device and install that. Check the XDA Forums to see if your device is included on their site. IF so, go into the development sub-section for your device to search for ROMs.