Is there a way to limit max RAM usage by Blender?
You cannot do inside blender, but if you are on windows you can check if these software can help you:


Process Tamer :
https://www.donationcoder.com/Software/Mouser/proctamer/
Process Lasso : https://bitsum.com/