Lyttelton was the third son of Sir Thomas Lyttelton, 4th Baronet, by his wife Christian, daughter of Sir Richard Temple, 3rd Baronet of Stowe, Buckinghamshire.
Almost immediately (13 August 1743) he was instituted to the rectory of Alvechurch, Worcestershire.