Only break object when colliding with another?
If you're unfamiliar with masking, read up what you can for your particular version of Blender.  See Here for a general description.
The example here is using a 3D mask and rendered in OpenGL but plain ol' vanilla 2D masks work just as well.

As for the cell fracturing, that was simply a cell-fractured cube the same size as the mask and hole.  It was placed inside the wall. A few particles were emitted from the same area for small debris effect.
The hole in the wall can be any shape you wish.  This one came from a subdivided cube.  It was given the 'Randomize' beauty treatment so it ended up jagged on all sides. (Tools panel)
That was then sliced in Edit mode so it ended up like a thick slice of bread, two flat walls (the bread) with it's crusts still jagged.

The flat front face was removed and used for the mask, the back one deleted and the jagged sides left as they were to give the hole it's depth.

The front face (removed) is moved to a separate layer which is designated to be the Masking layer.
Anything placed in there automatically becomes a mask and will cut a hole through the other layers .
This particular hole will obviously need it's walls but the mask is going to take them out.
We get around that by doing two renders to mix in the VSE.  One overlaying the other.
The 1st is called 'Top'.  That is alpha layered so the hole is rendered as transparent.
The 2nd is called 'Bottom' and that doesn't need (or want) an alpha layer, nor does it include the mask layer.  (An extra brick was inserted in the wall for effect)
.
When the two are mixed in the VSE, 'Top' on the upper strip, 'Bottom' underneath, we get the result shown at the top of this answer.
Note - the Top strip has to be set to &quot;Alpha Over&quot; in the VSE at right.  (Press N)