Can a linear reconstruction in compressive sensing perform well?
This is a good idea, I was pondering upon awhile ago. I drop some thoughts.

A Linear(no activations) MLP with a single hidden layer performs the
same or sometimes better than a Multi-layer model with ReLU
activations. What might be the cause of this?

The dimension reduction transform, i.e. taking CS measurements is a linear operation,  a measurement is weighted linear combination of input &quot;sparse&quot; vector. Are you reconstructing the same image that is used for training the network? The results could be different if the training image and test image are drastically different.

In the literature, to guarantee construction X has to be sparse in
some domain. But i tried removing all transformations from my model
and just worked in spatial domain and the model performs exactly the
same. shouldn't it struggle to reconstruct the signal?

There is actually no need to add that. Transformation is like any other function that network can &quot;learn&quot;. It does not matter if you map measurement to &quot;sampling&quot; domain or the transform domain.
Maybe you might be interested in seeing Baraniuk's work on similar thing as well:
https://patentimages.storage.googleapis.com/7d/cc/57/9f8ebb657d6fb7/US10985777.pdf