Annesley is said to have been born on April 15, 1715, in Dunmaine, County Wexford, to Arthur Annesley 5th Baron Altham (1689-14 Nov 1727) and his wife Mary Sheffield {1692-Oct 1729}, a daughter of John Sheffield, 1st Duke of Buckingham and Normanby [REF].
In 1740, after about 12 years working as an indentured servant, James escaped from the plantation (his third attempt) and made his way overland to Philadelphia where he took passage on a merchant ship to Port Royal Jamaica [REF].