PHP Fatal error: Allowed memory size of X bytes exhausted
Craft will run out-of-the-box with 32M of memory, no problem.  However, memory consumption will start to increase depending on several factors including:


The complexity of your site, including the number of fields, sections (and types of sections), categories, tags, users, entries, plugins in use (and how efficient they are working), etc.
More importantly, how your front-end templates are coded and how they are pulling in all of this data in.


Also worth noting that 32M is usually the default on older PHP installs (or really stingy newer hosts) and the latest PHP releases ship with something more reasonable, like 128M.

If you enable devMode, you can view memory profiling information in your browser's console to help you see where the memory is being distributed.