Background: Patients with frequent hospital readmissions, or high-utilizer patients (HUPs), are a major driver of rising healthcare costs in the United States. This group has a significant burden of medical illness, but less is known about whether or how social determinants of health may drive their increased healthcare use and poor health outcomes. Our study aimed to define the population of HUPs at a large, safety-net hospital system, to understand how these patients differ from patients who are not HUPs, and to analyze how their demographic, medical, and social factors contribute to their healthcare use and mortality rates. Methods: For this case-control study, data were collected via retrospective chart review. We included 247 patients admitted three or more times in a single calendar year between 2011 and 2013 and 247 controls with one or two admissions in a single calendar year matched for age, sex, and year of high-utilization. We used multivariable logistic regression models to understand which demographic, clinical, and social factors were associated with HUP status, and if HUP status was independently associated with mortality. Results: The factors that contributed significant odds of being a HUP included having Medicaid (OR 3.34, 95% CI 1.50, 7.44) or Medicare (OR 3.39, 95% CI 1.50, 7.67), having a history of recreational drug use (OR 2.44, 95% 1.36, 4.38), and being homeless (OR 3.73, 95% CI 1.69, 8.23) The mortality rate among HUPs was 22.6% compared to 8.9% among controls (p < 0.0001). Conclusions: These data show that social factors are related to high-utilization in this population. Future efforts to understand and improve the health of this population need to incorporate non-clinical patient factors.