We report the case of an immunocompetent man who presented with fever and abdominal pain and was found to have a hepatic abscess and a peri-renal abscess with a computerized tomography scan. The hepatic abscess was drained percutaneously and cultures revealed the presence of Lactococcus lactis that was sensitive to penicillin. The patient was successfully treated with ceftriaxone and metronidazole with resolution of the abscesses. Further work-up revealed atrophic gastritis, vitamin B12 deficiency, periodontitis and gingivitis, suggesting a possible site of entry for the development of the abscesses.