When he died in 1936 at Letterman Hospital in the Presidio of San Francisco at the age of 86, Varnum was the last surviving officer of those who had participated in the Battle of the Little Bighorn. He was buried in the San Francisco National Cemetery.