A native of Fredericksburg, Virginia, Gouldman was the son of John F. Gouldman Jr. (who served on the city council for 35 years) and his wife, the former Mabel Birdsall.
He married Margaret Elizabeth Moss, and they had a daughter Barbara Anne.