Boolean modifier with complex object (hole punching) makes blender go on its knees
I think you need to try out a specific add-on for your task. I have seen one called Box Cutter and its demoed on youtube, it probably has better code to handle the boolean operations than standard blender, lets hope.