determining timeouts / retry attempts in distributed systems
The simplest solution I have found to this problem is using rolling percentile latency. The optimal timeout will shift based on hardware, network configuration and the state of the operating system. Therefore it cannot be fixed/established beforehand, but approximated at runtime. The typical practice in industry is to maintain P95 and P99 response times. These are durations in which 95% and 99% of requests complete respectively over some time duration. So you could perhaps store the last hour of request times, and if the current request is longer than 95% of the others in the last rolling 60 minutes, then you deem that a timeout.
This will adapt and gradually become more strict if response times are fast, and more lenient if they become slow.
You can read more about percentile latency here.