Noisy parallel digital image capture
So it was actually an issue with my memory controller that was solved by decreasing the memory clock frequency.
I thought the memory was working totally cleanly because I had used it on a previous project and it responded well to various gradient test patterns. I was able to see that there was some sort of issue by changing to a test pattern that was more aggressively railing between black and white.
I was really thrown off by that noise following the contours of what was in the image, that really suggested sensor rather than memory in my mind.