Swinburne was the heir of Thomas Swinburne of Gunnerton, who had died before 1332, when Robert was around five years old.
Before August 1357, he married Agnes Felton, the daughter of the MP, William Felton.