 Bulwer was born on 25 May 1803 to General William Earle Bulwer of Heydon Hall and Wood Dalling, Norfolk and Elizabeth Barbara Lytton, daughter of Richard Warburton Lytton of Knebworth House, Hertfordshire.He had two older brothers, William Earle Lytton Bulwer (1799–1877) and Henry (1801–1872), later Lord Dalling and Bulwer.His father died and his mother moved to London when he was four years old.