IDFT - influence of sample numbers
How are you changing the number of samples ? If you are just truncating the 2048 samples $X$ to 1024 samples $X^t$, then the samples won't be complex conjugate anymore. 

First sample of the 2048 length sequence will be considered as DC and then the remaining should be conjugate symmetric around $N/2$. That means, $X[k] = X^*[2048-k], \forall k \in \{ 1, 1023 \}$ and $X[1024]$ must be real. Then IDFT will give real $x[n]$.

When you want to downsample it to 1024 samples, then, first of all there will be aliasing in time -domain but still you can get real time domain samples if the downsampled 1024 length $X^t$ maintains the above explained conjugate symmetry.