Removing years old grout from drywall in kitchen
Personally I'd just scrape it off with a putty knife.  If there are any dig and deep gouges after that step repair with spackle.  Then give it a skim coat of drywall compound, sand, and repaint.

Basically what you suggested except not actually new drywall, the steps are the same tho.