speeding up matrix corruption
My guess is that the bottleneck is drawing random numbers at each iteration.

Why guess when you can measure? I changed the height to 100 so the whole thing takes me about 8 seconds, reasonable to play around with. Removing the assert line on the bottom makes it a bit faster. Also removing the dirty = [j if...] line makes it a lot faster. As you'd expect from that inefficient line.
Not sure why you're making it so complicated. I think clean[idx] = 0 does the deletions you want, and it's fast.