Can someone help me understand smart group caching so I can prevent these deadlocks?
I cant answer everything but I think I can help clarify a few things;


  How can there be a difference between when the cache is flushed and when it is built, anyway? Surely refilling the cache means first you flush it, then you fill it. They cannot be separate


The answer to this lies in the state of the cache, there are in fact two states;


Current - does not need clearing or rebuilding
Invalid - this is either empty (as an operation flushed it or expired due to time/cron)


Important bit of information is that an action may chose to flush the cache without rebuilding it - as its not needed by that action but the action would affect it. A good example is a new contact being created - the ACL may be affected but in the create action the cache only needs to be flushed so that the ACL action can then build it.


  And for one last extremely confusing thing: I turned off the cron job. Why is the ACL cache still trying to update?


All caching has a expiration time, regardless of the cron if that time expires the next action will try to rebuild the cache. The idea of the cron is to carry out that action offline therefore reducing the time taken on the user action.

I'd like to work through the issue with you so please do keep this conversation going - we have some large customers running ACL without major cache issues so it should be a good case to establish exactly whats going wrong.

Perhaps you could share some numbers i.e. How many ACL groups/rules, smart groups, size of contacts etc. All of these factors will play in important part of the diagnosis.