Woolsey was born on the Breeze Hill plantation in Aiken, South Carolina, as Elizabeth (Elsa) Gammell Woolsey, but in later years took her middle name, which she shortened to "Gamel" (a Norse word meaning "old").
Gamel's mother, Bessie Gammell, was her father's second wife.