 Although his father was tied to the Howard family of England, his relations with the mainland quickly deteriorated.As a child he attended Brookeville Academy, an institution founded in 1808 by his ancestors.During the Civil War, Robertson, being a Quaker, refused to join either side.