Herbert was the sixth and youngest son of Henry Herbert, 1st Earl of Carnarvon by Elizabeth Alicia Maria, elder daughter of Charles Wyndham, 2nd Earl of Egremont.
He married, on 2 August 1830, Marianne, sixth daughter of Thomas Lempriere of La Motte, Jersey; she died on 7 August 1870.