 On 30 April 1783, he married Agnes Murray Elliot, daughter of Andrew Elliot, at Edinburgh and had by her ten daughters and two sons.Carnegie died at Gloucester Place in London and was buried at St Martin-in-the-Fields.He was succeeded in his titles by his older son James.