Cloth passes through Rigid Body
When I asked to simplify the blend, I ment to keep one example of cloth, only bricks that directly collide with cloth, there is no need to keep brick with a huge amount of vertices for this issue, so Decimate modifier could be applied (it slows down calculation any time you run simulation). You have also set Fake User for many objects. When deleted from a scene, they are still a part of file. To get rid of them you have to go into Outliner &gt; Ophan Data &gt; uncheck all of them and hit Purge. If objects are still there you can select them manually with Shift and with right-click choose Delet. Like that you can go from 184 MB to just 4 MB for a blend you could share with us (saved with compression enabled).


Collision
If you set Rigid Body just for colision with cloth (and you don't plan to destroy a wall later), it would not work.
You have to enable Collision that you can find above Cloth Simulation.

Tips:

If you plan just collide a cloth with a wall I would suggest you to use a simple plane (subdivided a few times), that is much faster for collision calculation.
To deactivate Rigid Body sim for all bricks at once - go to Scene Properties &gt; Rigid Body World &gt; Remove ... (it will remove all rigid body in a scene).
Quite annoying is cached cloth sim, that is not always updated when you change parametr. To recalculate sim with actual setup, I know only one way - deactivate&amp;activate monitor icon on Cloth button.
To apply modifier for all selected objects, search for Convert to Mesh. Sounds weird, right? It is a mesh already, but this operator apply all transformations at once for all selected.