Milk moved from New York City to live in San Francisco in 1972 . This was a time when lots of gay men moved to live in the Castro District of San Francisco . Milk moved from New York City to settle in San Francisco in 1972 amid a migration of gay men to the Castro District .