James Stewart was the fourth son of Sir James Steuart of Coltness (1608-1681), a banker in Edinburgh and Lord Provost of Edinburgh, and Anne Hope, niece of Sir Thomas Hope.
He was the brother of Sir Robert Steuart, 1st Baronet of Allanbank (1643-1707) and Sir Thomas Stewart of Coltness, 1st Baronet.