cache_menu items are huge Drupal 7.24

I have been stonewalled by an interesting problem on a D7 site. Essentially the cache_menu table was getting very large but not just in items (6300 odd) some of the items are enormous.

Its a new record for me, one of the items is 1.24 mb and 26 of them are between 800k and 1mb.

The site went down when it was being git with slightly elevated traffic, its running on a load balanced AWS cluster with RDS at the back, there's no lack of resources for a 20 simultaneous connections site.

i truncated the cache_menu and regenerated everything by browsing the site and all but a few of the items are gone now but there are still the 1.2mb one and a handful of the 800kb ones.

I followed this link -  http://drupal.org/node/1234830 from here Huge cache_menu table and it appears that this particular issue has been resolved for 7.13+ so i am thinking its something architectural going wrong.

There's no core hacks, we've got an impressive search system using a logic server network (running on SOLR externally) that does hook into taxonomy terms. The site is essentially a portal of links.

Could someone give me some pointers on where to start track this issue down?

edit: CID/sizes in cache_menu, query:

SELECT cid, LENGTH(`data`)/1024 as `size KB` FROM cache_menu WHERE LENGTH(`data`)/1024 &gt; '300' ORDER BY `size kb` DESC;


Result

links:management:tree-data:en:ec99d3452fef1ede622e66c68ba908b1dad455aa71f5e68648aeec6488b89c88
1094.2188
links:main-menu:all-cid:0:1:en
839.4873
links:main-menu:tree-data:en:ec99d3452fef1ede622e66c68ba908b1dad455aa71f5e68648aeec6488b89c88
815.6230
I suspect that you have some very large menus in your site.  Rebuilding menus is a an expensive operation, and menu trees can vary page-to-page, user-to-user, etc.  As a result, that data gets cached in {cache_menu}.

My suggestion is to query in your favorite SQL tool, and print_r() the data column for the large entries to see if anything looks out of the ordinary.  In general, the data will be your menu tree with everything from the various hook_menu() entries added in.  Modules that add extra data onto menu entries (like Menu Attributes) will have there data here, too.

You need to profile to see where the problem is actually happening.

It could be the result of a less-than-ideally configured MySQL instance.  If you do have 1M cache entries, then you may want to bump up the query cache result set limit (I think this defaults to 128K).

You may also want to investigate using memcache or redis as your cache backend.

However, if you really do have huge menus, then the slowdown could be from unserializing all of that data.  If this is the case, then you need to figure out a strategy to cache the menu output when your pages are being built up.  Enabling block level caching may help, or you may need to make your own menu blocks, and cache that output yourself.  Panels caching may be worth investigating.

It may sound odd, but you may also want try to disabling caching for {cache_menu}.  The easiest way to do this is to install one of the dummy cache modules, and configure it to just use that cache backend for {cache_menu}.