Duncan was born in Oakland, California, as Edward Howard Duncan Jr. His mother, Marguerite Pearl Duncan, had died in childbirth and his father was unable to afford him, so in 1920 he was adopted by Edwin and Minnehaha Symmes, a family of devout Theosophists.
At age three, Duncan was injured in an accident on the snow that resulted in his becoming cross-eyed and seeing double.