Token's available "Replacement patterns" nested table should only be displayed on-demand
The default token tree can contain many thousands of lines of HTML, and the corresponding Javascript listeners and functionality can totally break things memory wise (or just give you a darn laggy experience). Plus it can confuse the heck out of site administrators.

These issues refer to current developments for loading the token tree 'on demand' or via a popup:  


http://drupal.org/node/1684984 
http://drupal.org/node/1229568


And OP pointed out in his comments, the Token tweaks module lets you set a depth limit for the token tree, which seems to have solved his memory problem.