Nonlinear increase in logic utilization for FPGA design
This smells like something could be wrong with your user logic that results in a lot of it being optimized away.  4096 of anything should take up a LOT of logic resources.  Does the 4096 element design function correctly on the actual FPGA?