 As one of the Mitfords of Mitford Castle he was a cousin to Admiral Robert Mitford the bird-artist and Philip Meadows Taylor (author of "Confessions of a Thug"), and distantly related to Lord Redesdale (attorney general), William Mitford (historian), the Reverend John Mitford, and Mary Russell Mitford (author of "Our Village").The younger son of a younger son, Mitford had to make his own way in life and he chose a career in the navy.The patronage of his relative Lord Redesdale secured him a place as a midshipman on the in April 1795.