 Williams was born James Rainey in York County, South Carolina, on the plantation of Samuel Rainey about from Yorkville.He escaped from the Rainey plantation during the Civil War and joined the Union Army, where he eventually fought under General William Tecumseh Sherman in Georgia and the Carolinas.After the war, Rainey changed his surname to Williams and returned to York County and married a woman named Rose.