Atypical visual perceptual skills are thought to underlie unusual visual attention in autism spectrum disorders. We assessed whether individual differences in visual processing skills scaled with quantitative traits associated with the broader autism phenotype (BAP). Visual perception was assessed using the Figure-ground subtest of the Test of visual perceptual skills-3rd Edition (TVPS). In a large adult cohort (n = 209), TVPS-Figure Ground scores were positively correlated with autistic-like social features as assessed by the Broader autism phenotype questionnaire. This relationship was gender-specific, with males showing a correspondence between visual perceptual skills and autistic-like traits. This work supports the link between atypical visual perception and autism and highlights the importance in characterizing meaningful individual differences in clinically relevant behavioral phenotypes.