Difference boolean modifier leaving interfaces behind
There are probably precision problems where the compound cuboid components are not truly flush in a numerical sense.

You could try unioning the original cuboids into a single object rather than just joining them.

Alternatively, it looks like you could create the compound object via simple extrusions rather than as multiple cuboids.  This would give you a more consistent form that you could use for boolean operations.