In the early 1950s, Patterson became involved with the Russell Betterment Association (RBA), which was formed to combat the rampant vice and corruption occurring in Phenix City and Russell County.
He obtained a plurality in the 1954 Democratic primary, staging a runoff in May with Lee "Red" Porter of Gadsden.