Hugh Seymour was born in 1759 into one of the wealthiest families in England, as the fifth son of Francis Seymour-Conway, 1st Marquess of Hertford, and his wife Isabella Fitzroy (Hugh retained the surname "Seymour-Conway" until his father's death in 1794, at which point he shortened it to Seymour).
Following the Peace of Paris in 1783, Seymour took a house in London with his brother Lord George Seymour and John Willett Payne.