Infants with neonatal hypocalcemia often present with seizures, and neonatal hypocalcemia can be due to parathyroid (PTH) insufficiency or resistance. Causes of hypocalcemia with PTH elevation include increased phosphate load, vitamin D deficiency (VDD) or defects in metabolism, renal dysfunction, hypomagnesemia, genetic mutations resulting in end-organ resistance to PTH, or critical illness. Hypocalcemia has also been shown to be associated with Gram-negative bacteremia and sepsis in adults. We present the case of a full-term, formula-fed newborn presenting with late-onset hypocalcemic seizures and VDD in the setting of Klebsiella pneumoniae bacteremia. This case highlights that newborns presenting with hypocalcemic seizures should undergo a workup for sepsis.