Background. Excessive alcohol use is associated with brain damage but less is known about brain effects from moderate alcohol use. Previous findings indicate that patients with severe mental illness, particularly schizophrenia, are vulnerable to alcohol-related brain damage. We investigated the association between levels of alcohol consumption and cortical and subcortical brain structures in schizophrenia and bipolar disorder patients and healthy controls, and investigated for group differences for this association. Method. 1.5 T structural magnetic resonance images were acquired of 609 alcohol-using participants (165 schizophrenia patients, 172 bipolar disorder patients, 272 healthy controls), mean (S. D.) age 34.2 (9.9) years, 52% men. Past year alcohol use was assessed with the Alcohol Use Disorder Identification Test -Consumption part (AUDIT-C). General linear models were used to investigate associations between AUDIT-C score and cortical thickness, surface area, and total brain and subcortical volumes. Results. Increasing AUDIT-C score was linearly associated with thinner cortex in medial and dorsolateral frontal and parieto-occipital regions, and with larger left lateral ventricle volume. There was no significant interaction between AUDIT-C score and diagnostic group. The findings remained significant after controlling for substance use disorders, antipsychotic medication and illness severity. Conclusion. The results show a dose-dependent relationship between alcohol use and thinner cortex and ventricular expansion. The findings are present also at lower levels of alcohol consumption and do not differ between schizophrenia or bipolar disorder patients compared to healthy controls. Our results do not support previous findings of increased vulnerability for alcohol-related brain damage in severe mental illness.