Matlab - Constructing a Cosine wave with IFFT
You must keep in mind that for a real-valued signal, second half of your spectrum is a complex conjugate of all values below the Nyquist frequency. In your case:

X(2) = -32
X(N) = 32


as you can see the coefficients are not the complex conjugate of each other. Because of that you are ending up with round-off errors, since two frequency components are cancelling each other out.

Here is some modification to your code:

clc, clear all, close all
fs = 32;
dt = 1/fs;

N=64;
X=zeros(N,1);
X(2)=32;
X(N)=conj(X(2));
s=ifft(X)';

td = linspace(0, (length(s)-1)*dt,     length(s));

stem(td, s, 'b', 'linewidth', 2)
grid on
legend({'reconstructed signal'})


Which gives: