Related to Attorney General and politician Lord Redesdale, who became a patron, and to the historian William Mitford, he was born at Richmond, Surrey, on 13 August 1781.
After a brief experience as clerk in the army pay office, Mitford on 6 March 1801 matriculated at Oriel College, Oxford, under the tutorship of Edward Copleston, with Reginald Heber as a close friend, and graduated B.A. on 17 December 1804.