How to determine appropriate lowpass filter for Sigma-Delta modulator?
There will be at least three contributors to the error signal as you define it

a) in band distortion of the passband signal, caused by the filter cutting off too early

b) inadequate filtering of the out of band noise that the converter adds to the signal

c) in band distortion and noise added to the signal by the converter itself

A sigma delta converter produces out of band noise, that rises at a rate defined by the order of the converter. You want the reconstruction filter to have at least the same order as the converter, preferrably one order higher. Having it higher order will render far out noise irrelevant, and only close to band edge noise will be significant. As converter noise rises slowly, there's little need to use elliptic filters, conventional all pole filters will be just fine.

Assuming your converter is adequate for the job (which means the noise added does indeed fall outside your bandwidth of interest), then a filter that passes the passband, and rolls off faster than the converter noise rises, will also do the job.

If your converter is inadequate, then no amount of filtering will rescue it.

Your detailed definitions of x(t) and y(t), sample rates, any other filtering, might affect the detail of an optimum filter, but passing the passband while stopping the stopband is about all one can say without that detail.