Born in rural Victoria, Australia, Harvey moved to the suburbs of Melbourne in his childhood.
Harvey, his elder brother Philip, and younger brother Sebastian all attended the private boys school Caulfield Grammar School.