Smith was born to Joe Smith Sr. and Louise Lindsay Smith in the unincorporated community of Selma in northern Grant Parish, located south of Georgetown on U.S. Highway 165.
Smith graduated from Bolton High School in Alexandria, at the time the only high school for white youngsters in the Alexandria-Pineville area.