Robertson was born in Woodford County, Kentucky, the son of Scottish immigrant Cornelius Robertson and his wife Clarissa Hill (Keech) Robertson.
He and his wife Mary had three children, one of whom died in infancy.