Using sinc as a filter
You messed with f0/fm relationship. Try this code:

D = 5;
f0 = 1e3;
fm = 40e3;
t = 0:1:2*fm*D-1;
x = cos(2*pi*t*f0/fm);
h = sinc(-D*pi:1/10:D*pi);
x_p = filter(h,1,x);
plot(x_p); plot(x); plot(h);
figure; psd(x,2^11,fm);
figure; psd(h,2^11,fm);
figure; psd(x_p,2^11,fm);