Fixed point channel model for OFDM system
The effect you're observing is most certainly due to quantization errors. If your calculation of noise power was wrong you would get an offset between the simulated and the theoretical curve but not an error floor. The fact that it works fine with 32 bit resolution supports that hypothesis. You can analyze the output signal of your VHDL transmitter in order to find out whether it's already corrupted. There are several things I would consider:


Calculate the BER without channel (back-to-back). It should be zero but from your diagram I guess it will be something around 0.02.
Plot the constellation diagram. You should see distinct points not "clouds".
Calculate the signal-to-quantization-noise ratio $\gamma_\mathrm{q}$. Let $a_{\nu k}$ be the complex value of subcarrier $\nu$ in OFDM symbol $k$ after mapping and $b_{\nu k}$ be the respective subcarrier value at the transmitter output (can be obtained by applying FFT to the tx output). Then
$$
\gamma_\mathrm{q}=\frac{\sum_{\nu.k}|a_{\nu k}|^2}{\sum_{\nu.k}|b_{\nu k} - a_{\nu k}|^2}
$$