Elgin was born in Broomhall, Fife, the second son of Charles Bruce, 5th Earl of Elgin and his wife Martha Whyte.
In 1791, he was sent as a temporary envoy-extraordinary to Austria, while Sir Robert Keith was ill.