Buff was born in Newton, North Carolina, and, starting at the age of 17 served in the U.S. Navy between 1944 and 1948.
Buff graduated from North Carolina State University's School of Design in 1955 and worked for the Raleigh School Board.