When he was young, Passer was very shy, and had a dream of becoming an actor, but consented to his father's wishes by attending J. Lauritzen's sea training school near Svendborg in 1944.
During the 1950s he formed a duo with his colleague and friend Kjeld Petersen.