 He was twice married, first in 1915 to the actress Doris Lytton, and later to Dorothy Fleitman, with whom he had one son, John, who was an actor and then a long-time news anchor for Scottish Television; he took his own life in 1992.Toye's elder brother, Francis, was a well-known critic and Verdi scholar.Their sister Eleanor's daughter became a principal soprano with D'Oyly Carte under the name Jennifer Toye.