Disable geocoding for contacts but not for events
Off the top of my head, I think geocoding like this is an all or nothing thing. You can manually add the lat/long to an event location, so assuming that you don't have a whole lot of events, this should be pretty much acceptable (and just leave geocoding turned off).

With that said, no real contact details are shared with Google/Yahoo. A street address is simply sent to the geocoder without any names, or anything, and the lat/long is returned. No personally identifiable info is shared.