Other commuters riding too close
Just wave him past, that's what I do. I don't draft and I'm not interested in someone drafting off me. Nor am I particularly interested in being polite to someone who is interfering with my ride or looking around worrying about them.

Waving someone past is a pretty universal hand signal I would think. I use it with cars, cyclists, and anything else.