Inconsistent ACL behaviour
Ugh. This kind of bug is really hard to diagnose. Good call on noticing that memcache caches this behavior.

Are you on some kind of host with really aggressive caching by default? WPEngine, for example, has caused big problems.

If not, you should report this on issues.civicrm.org. My hunch is that caching/retrieval of the query for visiting the contact summary happens before the ACLs for custom fields are checked.  

You may be able to more reliably test this by creating a user not in Role 1 and logging in as them.  I'd be curious if one or both of the following happen:
 - visiting the contact as the non-privileged user causes the privileged user to not see the fields the next time
 - visiting the contact as the privileged user causes the non-privileged user to see the fields the next time (if so, a bad security problem)