Passman was the son of sharecroppers from Franklinton, the seat of Washington Parish in southeastern Louisiana.
In 1929, having relocated to Monroe, he formed Passman Equipment Company, which was involved in the manufacture of commercial refrigerators and distributed hotel and restaurant supplies and electrical appliances.