Background: Cerebral small vessel disease (SVD) is the common cause of cognitive decline in the old population. MRI can be used to clarify its mechanisms. However, the surrogate markers of MRI for early cognitive impairment in SVD remain uncertain to date. We investigated the cognitive impacts of cerebral microbleeds (CMBs), diffusion tensor imaging (DTI), and brain volumetric measurements in a cohort of post-stroke non-dementia SVD patients. Methods: Fifty five non-dementia SVD patients were consecutively recruited and categorized into two groups as no cognitive impairment (NCI) (n = 23) or vascular mild cognitive impairment (VaMCI) (n = 32). Detailed neuropsychological assessment and multimodal MRI were completed. Results: The two groups differed significantly on Z scores of all cognitive domains (all p < 0.01) except for the language. There were more patients with hypertension (p = 0.038) or depression (p = 0.019) in the VaMCI than those in the NCI group. Multiple regression analysis of cognition showed periventricular mean diffusivity (MD) ( = -0.457, p < 0.01) and deep CMBs numbers ( = -0.352, p < 0.01) as the predictors of attention/executive function, which explained 45.2% of the total variance. Periventricular MD was the independent predictor for either memory ( = -0.314, p < 0.05) or visuo-spatial function ( = -0.375, p < 0.01); however, only small proportion of variance could be accounted for (9.8% and 12.4%, respectively). Language was not found to be correlated with any of the MRI parameters. No correlation was found between brain atrophic indices and any of the cognitive measures. Conclusion: Arteriosclerotic CMBs and periventricular white matter disintegrity seem to be independent MRI surrogated markers in the early stage of cognitive impairment in SVD.