cutting a hole in cube with subdivision surface modifier already applied
First things first. Boolean is not a good choice in my opinion to receive simple shapes like that.

If I understand you right, you only want a cube which has rounden edges with a rectangular whole in the middle.

This is how I would do it.
First create a simple Cube.



Than subdivide it twice



Delete the inner faces



Now select "Bridge Edge Loop" under the "CTRL +E" Sub Menu



Last but not least, round your edges



Now you can still add a subdivision Modifier to make it even more smooth.



To keep the whole sharp, select all edges and increase the Mean Crease to 1.



Now select the faces you want to be sharp and click Faces -> Sharp under the "Shading"-Tab