Ashley was the third child and second son of Anthony Ashley-Cooper, 7th Earl of Shaftesbury and Lady Emily Cowper, eldest daughter of Peter Cowper, 5th Earl Cowper and sister of William Cowper-Temple, 1st Baron Mount Temple.
On William Cowper-Temple's death in 1888, he inherited a 10,000 acre estate on the Mullaghmore peninsula in Sligo, based around Classiebawn Castle.