He was born in Dublin and was the son of Charles Graves, Bishop of Limerick, Ardfert and Aghadoe, and his wife Selina, the daughter of John Cheyne (1777-1836), the Physician-General to the British Forces in Ireland.
Alfred was educated in England at Windermere College, Westmorland and Trinity College, Dublin.