The only son of Walter Augustus Shirley, bishop of Sodor and Man, he was born at Shirley, Derbyshire, on 24 July 1828.
In June 1846 Shirley matriculated at University College, Oxford, but in the following year he migrated to Wadham College, where he had gained a scholarship.