Would this type of design be scale-able?
Something similar has been done with GDDR memory, but it turned out that this wasn't useful in practice outside very few narrow cases, and even those have vanished.
Graphics cards often had a dedicated XOR operation for rendering a cursor by inverting pixels, and those could additionally be addressed to rectangles by masking address bits.
For general purpose computing, CPU caches are far more useful, especially as some architectures implement prefetch instructions that allow you to load the next dataset into cache while the current one is being processed. The local cache has acceptable latency compared to the CPU clock, and on anything that isn't register-starved like Intel CPUs before the PentiumPro, the vast majority of calculations would be performed in registers without delay, and only the result written to memory.