 Stillman was born in 1952 in Washington, D.C., to Margaret Drinker (née Riley), from Philadelphia, Pennsylvania, and a Democratic politician, John Sterling Stillman, an assistant secretary of commerce under President John F. Kennedy (a classmate of Stillman's father at Harvard), from Washington, D.C. His great-grandfather was businessman James Stillman; his great-great-grandfather, Charles Stillman, founded Brownsville, Texas.Stillman grew up in Cornwall, New York, and experienced depression during puberty.