Grosvenor was born Robert Grosvenor, the son of Lord Hugh Grosvenor, sixth son and tenth child of Hugh Grosvenor, 1st Duke of Westminster by his second wife, the Honourable Katherine Cavendish.
He was educated at Eton College, an all-boys public boarding school in Berkshire.