Gram was born in Moss as the son of district stipendiary magistrate Paul James Reinhold Harald Gram (1818-1900) and Jensine Sophie Wulfsberg (1810-1902).
In August 1878 he married Antoinette Augusta Brodtkorb (1857-1938).