Are there any citation managers that track author-specific affiliations?
Depending on how you define citation or bibliography management software, you can get part of the way there in the LaTeX world. The biblatex package in conjunction with the biber program supports data annotations. With biblatex/biber the database files can be in a variety of formats, but the most common is probably a bibtex .bib file (yeah the LaTeX world is confusing). Assuming a .bib file, for each entry you would add a field author+an with a value similar to 1=University A; 2=University B; ...; N=University N to denote the affiliation of each author (I think it can even support a list of affiliations for each author). With biblatex and biber you can store the author specific affiliations in the data base and get access to them when citing, but this is only part of the way there.

I am not aware of a nice graphical front end for managing bib files that directly supports data field annotations. The database is a plain text file, so maybe a text editor counts ...

I am also not aware of any biblatex styles that utilize data annotations, so you would have to roll your own. While updating a style to print the annotations is relatively easy, at the user level you are pretty limited. J Doe at University A and J Doe at University B are going to be treated as the same person. There is no user level support for sorting based on author name and then affiliation (or even affiliation of the first author).