Big Views query and PHP memory limits
Sounds like your memory limit on your server is too low.

Have a look under Reports >> Status report what it is.
Default on most servers are 32M. This is too low.

If you do not manage your own server then go beg your server admin to increase it. I can't give you a number, best to increase in 32Mb steps until you are happy.
I normally set mine to 256Mb during development and drop it to 128Mb on fairly large sites (this is just a guideline what I do).

If your server admin does not want to change it then first answer would be to get a new provider, which I know is not always the correct route. But read here how you might be able to set it yourself. Increase PHP settings