Handling TOCTTOU with system calls
One of the simplest ways to avoid TOCTTOU issues is for your system calls to perform the check at time of use (within the system call) instead of trusting a prior check. If you can gracefully fail a check upon use, the caller can instead rely upon exception handling to avoid a check race or a vulnerability. This methodology will help with both long-running and short-running scenarios.

To further assist with short-running scenarios, you could implement transactions for your security-related system calls. Of course, this would require being architected in early on, and would influence the design of the system call APIs, but it would provide concurrent consistency across the system.