Here in the U.S., it is my experience that over 75% of adults I meet socially will volunteer that phrase or a variation upon learning that I am a mathematician. I find this frustrating, since almost nobody would brag about being bad at history or English or really any other subject. Normally I change the subject, or perhaps say they must have had a bad classroom experience, but that feels unsatisfying and turns the conversation in an ugly direction.

Surely some of the bright minds here on Math.SE have found a good response. Note that I'm not looking to alienate a new acquaintance, so let's keep it positive.