Joseph Gandy was the son of Thomas Gandy (1744-1814) and Sophia née Adams (1743-1818).
He married, in 1801, Eleanor Susannah Baptist née Webb (1773-1867), daughter of Thomas Webb and Catherine née Wiggington.