James Atlay was born in Wakerley, Northamptonshire, the son of Henry Atlay (Rector of Great Casterton) and Elizabeth Rayner Hovell.
In 1867, he refused the bishopric of Calcutta, but in the following year accepted the bishopric of Hereford, in succession to Renn Hampden.