How do I remove the white residue on fresh tile grout?
This is typical.  It's called "grout haze", and you might have to wipe it off a couple of times before it won't reappear.

You just need a clean, damp rag, and some scrubbing.  If you didn't take it off immediately, it can set up, so you might need to check your local hardware store for 'grout haze remover' if water and elbow grease isn't doing it for you.

Well, if it had been on the tile, it'd have been typical, but as Jeff pointed out, you said it was on the grout itself.  I've never done colored grouts, so I have no idea if it's typical or not ... and you wouldn't want to scrub at the grout to remove the haze, particulary when the grout's fresh (which is when you want to remove it from the tile).

Looking around on the internet, I found mention that leaving water-based grout sealer to evaporate could do it.  They recommended re-sealing, and wiping off any excess, but others mentioned it took a lot of elbow grease and solvents.  On another site, it was mentioned it could be too much water in the grout, and suggested vinegar or a stronger acid to clean, then resealing; a few people mentioned getting grout dye to cover it up, but you have to strip any grout sealer first.