Grey was the eldest of the seven children of Colonel George Henry Grey and Harriet Jane Pearson, daughter of Charles Pearson.
Grey left university with no clear career plan and in the summer of 1884 he asked a neighbour, Lord Northbrook, at the time First Lord of the Admiralty, to find him "serious and unpaid employment."