System Generator: How to know if my FPGA could have enough resources to perform a design
Go on Xilinx's DSP course - that'll give you a really good feel for how much resource various dsp-type operations take up, and then you'll be able to estimate in your head.

Otherwise, you'll just have to build it and see.  To get a quick idea, generate the NGC from the sysgen block and use FPGAOptim to show you which blocks are using how much of which resource.  It'll not be the final final answer, but within a few percent of the LUT count. (Full disclosure - I wrote FPGAOptim at work)