Does the tmp directory have to be shared between servers?
Yes, if you have multiple servers serving the same domain, then you need to ensure that the two Drupal server instances share the same filesystem. Best is to use NFS mount, so it is physically only one folder. You may get by with rsync in a cron depending on how the filesystem is used in your application.