We can learn about the affective content of the environment by observing the behavior of others; their responses to stimuli tend to be appropriate to the context. To investigate the impact of observing such appropriate, compared with inappropriate, behaviors, we developed a novel behavioral task where participants observed different faces reacting to emotional scenes. We found that affective categorization of a scene was facilitated when it was presented alongside an appropriate facial expression (Experiment 1). Further, we observed that several brain areas in the right hemisphere-the putamen, insula, orbitofrontal cortex, and dorsal anterior cingulate cortex-were more activate when viewing faces that were previously observed emoting inappropriately (Experiment 2). We contend that these areas form a network that codes for the retrieval of affective conflict information generated by observing individuals producing inappropriate emotions.