Low bone mineral density (BMD) and osteoporosis have become a public health problem. We found that non-Hispanic white, black, and Asian adults with extremely low education and personal income are more likely to have lower BMD. This relationship is gender-specific. These findings are valuable to guide bone health interventions. The evidence is limited regarding the relationship between socioeconomic status (SES) and bone mineral density (BMD) for minority populations in the USA, as well as the relationship between SES and BMD for men. This study explored and examined the relationship between SES and BMD by race/ethnicity and gender. Data (n = 6568) from the Louisiana Osteoporosis Study (LOS) was examined, including data for non-Hispanic whites (n = 4153), non-Hispanic blacks (n = 1907), and non-Hispanic Asians (n = 508). General linear models were used to estimate the relationship of SES and BMD (total hip and lumbar spine) stratified by race/ethnicity and gender. Adjustments were made for physiological and behavioral factors. After adjusting for covariates, men with education levels below high school graduate experienced relatively low hip BMD than their counterparts with college or graduate education (p < 0.05). In addition, women reporting a personal annual income under $20,000 had relatively low hip and spine BMD than their counterparts with higher income level(s) (p < 0.05). Establishing a conclusive positive or negative association between BMD and SES proved to be difficult. However, individuals who are at an extreme SES disadvantage are the most vulnerable to have relatively low BMD in the study population. Efforts to promote bone health may benefit from focusing on men with low education levels and women with low individual income.