Is there any way to cancel Panels' overlay edit windows?
I did some research and looked into Panels 3 code so far I don't see any nojs fallbacks. You can also check if there are fallbacks when you try to disable Javascript in the browser.

This may not be the answer on how to disable Panels modals but you may increase PHP, MySQL and or Apache settings and increase memory to your local host.