Is there a standard for maximum torque that should be used to fasten PCBs to an enclosure?
It depends most on what and how you are fastening. Many different materials will have different properties. 

For example, the compressive strength of FR4 material is at least 460MPa, or 460N/mm^2.

That number is one you want to stay well clear of, especially if you have multi-layer boards (since you don't know the exact specs of the prepreg your fab uses). So let's say "definitely don't go over 100N/mm^2".

Now here's the rub. That's the compression strength, but it does mean you have to have proper rings to prevent extra warping forces when the nut is tightened and even then some warping may still happen if tightened too far.

Then how that translates to a Nm reading depends on:


The surface area of the rings you use: Smallest surface determines the maximum number of downward Newtons.
The threading on the bolt; It's neatness and the number of threads per unit length.


Doubtless there's conversion charts for that on the internet for certain types of screws, but I couldn't find them just now (though I did only spend about 1 minute searching, since that's also what it took to get the compression strength of FR4).

All that said, I do think the tiny surface area of the threads on your screws, unless you use 13.9 grade metal, maybe, will be the weakest link in your system. The cheap DIY store screws will definitely break before up to 8 layers of PCB. So will, as I know from experience, RVS A2 screws.

That is, I did once split a 4 layer board with an M4 screw and two washers, but the RVS A2 one broke on hand-tightening, then I took out a military grade 13.9 one and needed a power wrench to tighten it enough to crack open.

But if you want to stay super safe, get the data of a brass spacer in the thread size you use and then apply the maximum allowable torque for that, even if you use harder types of fasteners, a metalised PCB hole will certainly survive quite well. Brass spacers are relatively weak compared to FR4 compression strengths and usually with brass spacers you can still tighten stuff well enough to mount PCBs and such.