Is there any way to permanently crop a PDF without decreasing its quality?
Cropping (in Acrobat) is non-destructive, as you noted.

If you really want to permanently cut away parts of your scan, it would be better to open the file in an image processing software, do the cutting away (aka cropping) there and resave as PDF (or TIFF, and then bring back to Acrobat for a PDF).

If you do not change the image resolution, the image quality will remain.