At the time of the song's initial popularity in the 1950s, many believed it was inspired by a green-doored restaurant and bar called "The Shack" in Columbia, Missouri, where singer Jim Lowe had attended the University of Missouri.
An oft-repeated urban legend has developed saying the song refers to London's first lesbian club, Gateways (1930-1985), which was in Bramerton Street in Chelsea.