Painting old plaster walls
I have done some of this same work.  I have heard that glue residue can mess up the paint finish over time, so I wanted to make sure that was taken into account when selecting a primer.  

I called the Killz company (Killz.com) because that's the brand that's easiest to find in my area and I have used their products in the past with good results.  

Their tech support told me that Killz Klear (or Killz Klear Max) is the best product for covering up adhesive residue.  The rep also said that Klear is good for chalky surfaces in general, so it sounds like it's a good choice on unpainted plaster even if you do manage to remove all the adhesive.  

The Killz Klear is water based and should only be used with latex paints.