Dudley was the illegitimate son of Edward Sutton, 5th Baron Dudley of Dudley Castle, and grandson of Edward Sutton, 4th Baron Dudley.
Lord Dudley (though he had a legitimate son, and a granddaughter by him, as well as four legitimate daughters and numerous grandchildren) seems to have attended to the up-bringing of his natural children by Elizabeth Tomlinson; he educated and provided for them.