Why the mirror modifier ruins the boolean shape
If you take the mesh with only the mirror, you can notice that the mirror will not merge the parts:

Here in the top part of the image, the mesh before the mirror is applied and on the bottom, after its application (both parts are not merged).



As they are not merge, Blender can only see opened meshes, so the boolean differences operate on open meshes too and that's why the result is opened too.

To fix that, just move the empty a bit along X.

Secondly, and for the same reason, the mirror needs to be set on top of the stack (at least before the booleans), as if it is not, when a boolean operates, the mesh is opened and you have the result shown in your question.

So the best is to set the mirror on top, then the subdivision and last the booleans:

Here is the result (after the empty is moved a bit along +x and the modifiers reordered: