 Fisher was born in Sussex County, Delaware, into a Quaker family with historic roots, growing up in Lewes, Delaware.His father, Thomas Fisher (1669–1713), and mother, Margery Maud (1671–1770), were both Quakers, as were his grandparents.Margery Maud was a stepdaughter of Dr. Thomas Wynne.