How do I NOT clear memcache or other cache tables?
One setting you can play around with is the minimum cache lifetime setting. It will help, but it won't solve the issue.

The standard Drupal cache doesn't allow for what you are describing. One can do some exotic stuff to update caches in the background but unless you have a large developer budget to make this work I would move on. Note: The 6.x version of boost is the only thing I know of that allows for the cache to be updated in the background.

If you are looking to build a spider, http://drupal.org/project/httprl/ is the way to go. I hope to be developing with this more in the future.