The line "I hear you went up to Saratoga and your horse naturally won" refers to the Saratoga Race Course horse-racing season held in late July, August, and early September in Saratoga Springs, New York.
Simon said the line "clouds in my coffee" came "from an airplane flight that I took with Billy Mernit, who was my friend and piano player at the time.