Using structural MRI, we investigated the brain substrates of both affective and cognitive theory of mind (ToM) in 19 patients with semantic dementia. We also ran intrinsic connectivity analyses to identify the networks to which the substrates belong and whether they are functionally disturbed in semantic dementia. In line with previous studies, we observed a ToM impairment in patients with semantic dementia even when semantic memory was regressed out. Our results also highlighted different neural bases according to the nature (affective or cognitive) of the representations being inferred. The affective ToM deficit was associated with atrophy in the amygdala, suggesting the involvement of emotion-processing deficits in this impairment. By contrast, cognitive ToM performances were correlated with the volume of medial prefrontal and parietal regions, as well as the right frontal operculum. Intrinsic connectivity analyses revealed decreased functional connectivity, mainly between midline cortical regions and temporal regions. They also showed that left medial temporal regions were functionally isolated, a further possible hindrance to normal social cognitive functioning in semantic dementia. Overall, this study addressed for the first time the neuroanatomical substrates of both cognitive and affective ToM disruption in semantic dementia, highlighting disturbed connectivity within the networks that sustain these abilities.