Can tile glue for wall tiles contain asbestos?
Yes you are definitely limiting your exposure by removing the drywall and NOT SCRAPING.

Even removing the Drywall - asbestos dust can fly around.


Get a disposable suit so that you don't get any fibers on your clothing.
Get a good respirator suitable for asbestos protection.


I am not affiliated in any way with the below link but providing as an example

Or buy the Kit not expensive at all - reasonably priced:
https://www.pksafety.com/lead-asbestos-removal-pro-kit.html