Removing wallpaper on concrete wall and prepare for paint
I think you have the right idea. Wall paper is a huge pain to remove correctly, and that glue will absolutely cause problems with future coatings if it's not dealt with.
I have 2 suggestions.

If you can muster the willpower to clean the walls one more time, get some TSP (trisodium phosphate, the real stuff, not the &quot;TSP Substitute&quot;) and use it mixed with hot water to really scrub, then rinse with water. It does a good job of eating up that gunk.
Use an oil primer. As good a product as Gardz is, you absolutely must avoid any kind of water based product being directly in contact with the remaining paste residue. The water can re-activate it and cause poor adhesion and peeling. If that base coat doesn't stick, anything else that goes on top of it will fail sooner or later. Get a respirator with an organic vapor cartridge, a can of mineral spirits... and maybe a couple of fans. It's kind of a pain, but it dries fast. It's really not that bad.

Also, @JACK makes a good suggestion in the comments to deal with the texturing, but I will add a warning: don't mud directly on top of the unprimed wall, for the same reason I recommended not using a water based primer.