Previous studies have suggested that real-time person perception relies on continuous competition, in which partially active categories smoothly compete over time. Here, two studies demonstrated the involvement of a different kind of competition. In Study 1, before participants selected the correct sex category for morphed faces, their mouse trajectories often exhibited a continuous attraction toward the incorrect category that increased with sex-category ambiguity, indicating continuous competition. On other trials, however, trajectories initially pursued the incorrect category and then abruptly redirected toward the correct category, suggesting early incorrect category activation that was rapidly reversed later in processing. These abrupt category reversals also increased with ambiguity. In Study 2, participants were presented with faces containing a sex-typical or sex-atypical hair cue, in a context in which the norm was either sex-typical targets (normative context) or sex-atypical targets (counternormative context). Sex-atypical targets induced greater competition in the normative context, but sex-typical targets induced greater competition in the counternormative context. Together, these results demonstrate that categorizing others involves both smooth competition and abrupt category shifts, and that these flexibly adapt to the social context.