Removing Wallpaper Glue - All of it or can I leave faint bits behind?
I also think removing as much old adhesive as possible is what you want to do. I made the mistake of not removing it all a few years ago and now I have to go back. Also, seal with kitz oil based primer.  The skim coat sounds like a lot of work to get it smooth. Is the wall that rough? Af for the need to prime, definitely after skimming, or on any holes you patch. I wouldn't prime before skimming unless there was still glue left.