How can I center a drill bit so that it drills absolutely straight down rather than perhaps a little off to the side?
Update 5/12/2013

While my jig below does work, this product is even better!

Big Gator Tools V-DrillGuide Standard Size Portable Drilling Guide



Original Answer

I am answering my own question here with an awesome, home-grown solution to my problem.  I fastened two pieces of wood together at a 90 degree angle.  Corner clamps made it easy to get it perfect.  I then took a third piece of wood and cut a 45 degree bevel off the corner.  This piece is my drill bit adapter that fits a number of larger-sized drill bits, but was made with my 3/8" bit in mind.  I also made another for smaller sizes (not shown).

To hold the adapter firmly in place and tight against the bit, I use one of my smaller c-clamps.  Using this custom guide, I was able to drill a near-perfect hole into the subject.  I also used another similar custom assembly that works with a plug cutter bit for my power drill.

No expensive equipment required!