Patients with schizophrenia show specific abnormalities in visual perception, and patients with bipolar disorder may have related perceptual deficits. During tasks that highlight perceptual dysfunction, patients with schizophrenia show abnormal activity in visual brain areas, including the lateral occipital complex (LOC) and early retinotopic cortex. It is unclear whether the anatomical structure of those visual areas is atypical in schizophrenia and bipolar disorder. In members of those two patient groups and healthy controls, we localized LOC and early retinotopic cortex individually for each participant using functional magnetic resonance imaging (MRI), then measured the thickness of those regions of interest using structural MRI scans. In both regions, patients with schizophrenia had the thinnest cortex, controls had the thickest cortex, and bipolar patients had intermediate cortical thickness. A control region, motor cortex, did not show this pattern of group differences. The thickness of each visual region of interest was significantly correlated with performance on a visual object masking task, but only in schizophrenia patients. These findings suggest an anatomical substrate for visual processing abnormalities that have been found with both neural and behavioral measures in schizophrenia and other severe mental illnesses.