 Colley was born in Klamath Falls, Oregon, to Muriel and Pete Colley.He attended Klamath Union High School and actively played football and track and field, which led to an unsuccessful try-out for the 1960 Summer Olympics.He later attended the University of Oregon and studied architecture.