 James Harris was born at Salisbury, Wiltshire, the son of James Harris (1674–1731) by his second marriage to Elizabeth (c. 1682–1744), daughter of Anthony Ashley Cooper, 2nd Earl of Shaftesbury.He was educated at the Salisbury Cathedral School, and Wadham College, Oxford.On leaving university he was entered at Lincoln's Inn as a student of law, though he was not intended for the Bar.