A member of the Shirley family headed by the Earl Ferrers, Shirley was the son of Evelyn Shirley and Mary Clara Elizabeth, daughter of Sir Edmund Lechmere, 2nd Baronet.
He was presented to the Prince of Wales (later Edward VII) at a special levée at St. James Palace on 1 June 1869.