Lewis was born to the poor family of Elmo and Mamie Lewis in Ferriday, in Concordia Parish, in eastern Louisiana, and began playing piano in his youth with two cousins, Mickey Gilley (later a popular country music singer) and Jimmy Swaggart (later a popular TV evangelist).
He was also influenced by the Great American Songbook and popular country singers like Jimmie Rodgers and Hank Williams.