Artifacts with boolean modifier
I expand my comments above, to post some example image. I still think it's a matter of mesh density:

To better visually describe I exagerated size of the diff meshes, but this is the overall setup:

 

this is how a subdiv level of 2 on a standard UV sphere, in boolean diff with a cube, enough subdivided and extruded, then hidden. Still, you see some artifact on the hole edge.

I tried a gif animation showing the effect of changing settings, but due to color banding it gets lost... so here is a rendered comparison with subdiv 0,1,2, and 4 (enough in this example)



As said above I can't show you here, but even "moving around" the sphere doesn't show artifacts (or they are much more subtle) in the viewport, and the same goes in the render.

I don't know how better you can solve this, but using much higher resolution on the edges of the subctraction, at least, will help blender to find a "smooth" setup for normals of geometry around that area...