How to disable cache refresh on Manage Groups page
The manage groups page does not force a refresh of the smart group cache but there is a button to do so on the right-hand side:


Without clicking that button, what it does do is count the number of contacts already in the cache, but this query has been optimized and tested against a very large (6 million group_contact records) database.

You'll find more information at https://issues.civicrm.org/jira/browse/CRM-15107