Lyttelton was born at Hagley Hall in Worcestershire, the second son of Charles Lyttelton, 5th Baron Lyttelton and later 8th Viscount Cobham, and Mary Susan Caroline Cavendish (second daughter of the 2nd Baron Chesham).
At Trinity, Lyttelton was a distinguished shot put competitor, winning the event for Cambridge v Oxford three years in a row (1904, 37'7"; 1905, 37'11" and 1906, 38'3¾") [REF].