The Carandinis, Lee's maternal ancestors, were given the right to bear the coat of arms of the Holy Roman Empire by the Emperor Frederick Barbarossa.
Lee was engaged for a time in the late 1950s to Henriette von Rosen, whom he had met at a nightclub in Stockholm.