CiviCRM does not have permission to write temp files
There's no single perfect solution to this, but there are several adequate solutions. To highlight some trade-offs among the approaches raised so far:


For each security principal, use separate data-directories (like bgm's answer).

Pro: Portable -- works in many different environments. Doesn't require any special tools or root permissions.
Con: Only works when data can be split apart by user. For example, it could potentially work for logs/caches (because they don't really need to be shared) but is more problematic for uploads/attachments (because both users should have read/write on the same objects).

Create a cronjob or daemon to adjust permissions (like Chris's answer)

Pro: Portable -- you can design the daemon or cronjob to build on the traditional POSIX UGO/RWX triplets. Works with all data (logs/caches/attachments).
Con: Requires installing/running a new root-permissioned service. Watch out for symlinks/hardlinks.

Use the operating system's ACL features (like amp's linuxAcl and osxAcl policies)

Pro: Works with all data (logs/caches/attachments). Doesn't require daemon or regularly running as root.
Con: Not standardized -- technical steps depend on the environment (OS/distro/filesystem). Sometimes applications try to be clever and override the permissions -- which breaks ACLs for specific files/use-cases.
(Note: amp tries to abstract this, but as with any abstraction, it's probably not worth the overhead unless you buy into the general vision.)