Agee was born in Knoxville, Tennessee, to Hugh James Agee and Laura Whitman Tyler, at Highland Avenue and 15th Street, which was renamed James Agee Street in 1999, in what is now the Fort Sanders neighborhood [REF].
At Phillips Exeter, Agee was president of The Lantern Club and editor of the Monthly where his first short stories, plays, poetry and articles were published.