Robert Stewart was born in Henry Street, Dublin, Ireland, in 1769 the son of Robert Stewart (1739-1821) of Newtownards and Comber in County Down, with properties in Counties Donegal and Londonderry.
Stewart's mother, who died in childbirth when he was a year old, was Lady Sarah Frances Seymour-Conway, daughter of Francis Seymour-Conway, 1st Marquess of Hertford (a former British Ambassador to France (1764-65) and Lord Lieutenant of Ireland (1765-66)) and Isabella Fitzroy, daughter of Charles FitzRoy, 2nd Duke of Grafton.