How do I remove construction glue from brick?
As the comment said, a heat gun may make scraping this off much easier. First, try scraping some off without a heat gun.  You might get lucky and it might "break" off the surface in large solid pieces.  If it does, the heat gun might make it more difficult.  If it does not come off easily, then I'd try the heat gun.

Start a low temperature and work your way up gradually until the adhesive softens up; you don't want to burn it, just soften it up. Please beware that heating this adhesive might release harmful vapors.  Make sure your area is well ventilated when doing this. You might even opt to wear a respirator. 

I would personally stay away from solvents in this case especially if you do not know what type of adhesive it is.