First impressions made to photographs of faces can depend as much on momentary characteristics of the photographed image (within-person variability) as on consistent properties of the face of the person depicted (between-person variability). Here, we examine two important sources of within-person variability: emotional expression and viewpoint. We find more within-person variability than between-person variability for social impressions of key traits of trustworthiness, dominance, and attractiveness, which index the main dimensions in theoretical models of facial impressions. The most important source of this variability is the emotional expression of the face, but the viewpoint of the photograph also affects impressions and modulates the effects of expression. For example, faces look most trustworthy with a happy expression when they are facing the perceiver, compared to when they are facing elsewhere, whereas the opposite is true for anger and disgust. Our findings highlight the integration of these different sources of variability in social impression formation.