Brown's interest in secrets and puzzles stems from their presence in his household as a child, where codes and ciphers were the linchpin tying together the mathematics, music, and languages in which his parents worked.
After graduating from Phillips Exeter, Brown attended Amherst College, where he was a member of Psi Upsilon fraternity.