Ifft through Matrix multiplication
You can alternatively create a DFT matrix in matlab using this code:

exp(-1j*2*pi* ((0:N-1)/N).' * (0:N-1))


And the IDFT matrix thus:

1/N * exp(1j*2*pi* ((0:N-1)/N).' * (0:N-1))


As the only difference betweenm DFT and IDFT is the sign and a scaling factor.

You could alternatively just do:

ifft(eye(N))


But this doesn't get around needing the full DFT matrix, and honestly I don't see an obvious way of perfoming a DFT by matrix multiplication without actually having the DFT matrix. Perhaps if you were writing your own code you could leverage the symmetry of the DFT matrix so you only needed half of it but I don't think this idea will get you anywhere in Matlab.