20,000 sleeping threads
Attempting to train a model in a multi-GPU scenario using a recent snapshot of master branch, and discovered that TF creates a huge number of threads, all of which sit idle and do nothing. The exact sequence of operations is as follows:

* A collective XLA op is executed.
* It calls ResolveDeviceAssignment() https://github.com/tensorflow/tensorflow/blob/master/tensorflow/compiler/jit/kernels/xla_ops.cc#L616
* Which calls BaseCollectiveExecutor::CompleteParamsAsync() with a timeout of 1000 seconds https://github.com/tensorflow/tensorflow/blob/master/tensorflow/compiler/tf2xla/xla_helpers.cc#L147
* Which calls SchedNonBlockingClosureAfter() https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/common_runtime/base_collective_executor.cc#L361
* Which creates a thread which sleeps for the specified period of time (1000 s) and then executes the supplied closure function https://github.com/tensorflow/tensorflow/blob/master/third_party/xla/third_party/tsl/tsl/platform/default/env.cc#L181

The problem is essentially that SchedNonBlockingClosureAfter() is implemented in an exceedingly wasteful way, because its author assumed it to be something used once in a blue moon, and now it's being called for every collective XLA op (i.e. multiple calls for every step during training).

I suspect that this massive pool of idle threads is what causes some of my training runs to crash after a while, with "terminate called after throwing an instance of 'std::system_error' / terminate called recursively /   what():  Resource temporarily unavailable".