Something wrong with my hole
You can also mix the material of the sides of the hole with a holdout shader while using the backfacing output of a geometry node as the fac input of the mix shader.

Without the holdout shader:



With the holdout shader: