How can I remove vinyl base without damaging the drywall?
Buy tall trim to cover up the glue issues or just skim coat the whole area.  You will spend more time tying to get this off clean then ripping and skim coating the area.