//tensorflow/dtensor/python/tests:input_util_test is flaky
Can you see what version of portpicker is installed?  @MichaelHudgins It should be 1.4.0 for this one as that is what should be installed in the docker container I think. But the log does not show it. https://source.cloud.google.com/results/invocations/3be5d51d-0531-4b39-8813-703e6e84c608/log You are right, the place i am thinking of it ended up getting removed from the docker files when the pip tests were changed.  Lets try and update and see if it helps.  Will give that a try shortly  Updated portpicker in https://github.com/tensorflow/tensorflow/commit/948361e05e8bbe0efe9e5f79867d17331f085eaf .  The updates have some improvements to the port picking logic so lets see if that helps out.  The ARM jobs are now failing due to a permissions issue that was shown up by the update to portpicker. This PR should resolve that https://github.com/tensorflow/tensorflow/pull/60705 so it would be good if that could get merged. The above mentioned PR https://github.com/tensorflow/tensorflow/pull/60705 is merged now, could you please let us know if that solves your issue. Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60671">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60671">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

git HEAD

### Custom Code

No

### OS Platform and Distribution

Ubuntu 20.04

### Mobile device

n/a

### Python version

3.9.16

### Bazel version

5.3.0

### GCC/Compiler version

9.4.0

### CUDA/cuDNN version

n/a

### GPU model and memory

n/a

### Current Behaviour?

Sometimes test fails.

### Standalone code to reproduce the issue

```shell
docker exec tf bazel --bazelrc=/usertools/cpu.bazelrc test --config=rbe --config=pycpp --config=build_event_export
```


### Relevant log output

```shell
[ RUN      ] DTensorDatasetTest.testIterPrefetchEnabled
I0523 01:36:42.332243 140069302650688 mesh_util.py:35] This is client 0 of 1 clients
I0523 01:36:42.332387 140069302650688 mesh_util.py:36] Number of global CPU devices: 16
I0523 01:36:42.332595 140069302650688 mesh_util.py:39] Global device IDs: [[[ 0  1]
  [ 2  3]]

 [[ 4  5]
  [ 6  7]]

 [[ 8  9]
  [10 11]]

 [[12 13]
  [14 15]]]
I0523 01:36:42.333161 140069302650688 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
I0523 01:36:42.333399 140069302650688 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15']
2023-05-23 01:36:42.502792: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization
2023-05-23 01:36:42.539630: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization
2023-05-23 01:36:42.571014: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization
2023-05-23 01:36:42.601962: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization
2023-05-23 01:36:42.628375: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_6' with dtype resource
	 [[{{node Placeholder/_6}}]]
2023-05-23 01:36:42.629052: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype int64 and shape [32]
	 [[{{node Placeholder/_4}}]]
2023-05-23 01:36:42.662961: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_6' with dtype resource
	 [[{{node Placeholder/_6}}]]
2023-05-23 01:36:42.663830: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype int64 and shape [32]
	 [[{{node Placeholder/_4}}]]
2023-05-23 01:36:42.697634: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_6' with dtype resource
	 [[{{node Placeholder/_6}}]]
2023-05-23 01:36:42.698468: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_6' with dtype resource
	 [[{{node Placeholder/_6}}]]
2023-05-23 01:36:42.731407: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype int64 and shape [32]
	 [[{{node Placeholder/_4}}]]
2023-05-23 01:36:42.732128: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype int64 and shape [32]
	 [[{{node Placeholder/_4}}]]
2023-05-23 01:36:42.765658: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype int64 and shape [32]
	 [[{{node Placeholder/_4}}]]
2023-05-23 01:36:42.766547: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_6' with dtype resource
	 [[{{node Placeholder/_6}}]]
2023-05-23 01:36:42.799498: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [3,1]
	 [[{{node Placeholder/_16}}]]
2023-05-23 01:36:42.800179: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_6' with dtype resource
	 [[{{node Placeholder/_6}}]]
2023-05-23 01:36:42.833451: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [3,1]
	 [[{{node Placeholder/_16}}]]
2023-05-23 01:36:42.834362: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype int64 and shape [32]
	 [[{{node Placeholder/_4}}]]
2023-05-23 01:36:42.867624: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype int64 and shape [32]
	 [[{{node Placeholder/_4}}]]
2023-05-23 01:36:42.868470: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype int64 and shape [32]
	 [[{{node Placeholder/_4}}]]
2023-05-23 01:36:42.901365: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [3,1]
	 [[{{node Placeholder/_16}}]]
2023-05-23 01:36:42.902052: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [3,1]
	 [[{{node Placeholder/_16}}]]
2023-05-23 01:36:42.934892: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [3,1]
	 [[{{node Placeholder/_16}}]]
2023-05-23 01:36:42.935557: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_6' with dtype resource
	 [[{{node Placeholder/_6}}]]
2023-05-23 01:36:42.969417: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_6' with dtype resource
	 [[{{node Placeholder/_6}}]]
2023-05-23 01:36:42.970324: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [3,1]
	 [[{{node Placeholder/_16}}]]
2023-05-23 01:36:43.005591: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype int64 and shape [32]
	 [[{{node Placeholder/_4}}]]
2023-05-23 01:36:43.006315: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [3,1]
	 [[{{node Placeholder/_16}}]]
2023-05-23 01:36:43.039225: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [3,1]
	 [[{{node Placeholder/_16}}]]
2023-05-23 01:36:43.039923: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_6' with dtype resource
	 [[{{node Placeholder/_6}}]]
2023-05-23 01:36:43.074097: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [3,1]
	 [[{{node Placeholder/_16}}]]
2023-05-23 01:36:43.074987: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_6' with dtype resource
	 [[{{node Placeholder/_6}}]]
2023-05-23 01:36:43.107878: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [3,1]
	 [[{{node Placeholder/_16}}]]
2023-05-23 01:36:43.108533: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_6' with dtype resource
	 [[{{node Placeholder/_6}}]]
2023-05-23 01:36:43.144069: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype int64 and shape [32]
	 [[{{node Placeholder/_4}}]]
2023-05-23 01:36:43.145070: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [3,1]
	 [[{{node Placeholder/_16}}]]
2023-05-23 01:36:43.438866: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:1] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled
	 [[{{node tf.StatefulPartitionedCall/eager_operation}}]]
2023-05-23 01:36:43.439513: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled
	 [[{{node tf.StatefulPartitionedCall/eager_operation}}]]
2023-05-23 01:36:43.440149: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:5] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled
	 [[{{node tf.StatefulPartitionedCall/eager_operation}}]]
2023-05-23 01:36:43.440610: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:4] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled
	 [[{{node tf.StatefulPartitionedCall/eager_operation}}]]
2023-05-23 01:36:43.441222: E tensorflow/dtensor/cc/dtensor_device.cc:2247] Error executing CopyToMesh {{function_node IteratorGetNext__func_14891566070902430035_8539663482787142021_8624502544051534713_3}} End of sequence
	 [[{{node tf.StatefulPartitionedCall/eager_operation}}]]
	Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors.
INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterPrefetchEnabled): 1.16s
I0523 01:36:43.496025 140069302650688 test_util.py:2464] time(__main__.DTensorDatasetTest.testIterPrefetchEnabled): 1.16s
[  FAILED  ] DTensorDatasetTest.testIterPrefetchEnabled
```
</details>