//tensorflow/python/data/kernel_tests:snapshot_test is flaky
@rishikasinha-tf flaky test AARCH64 log
https://github.com/tensorflow/tensorflow/actions/runs/5378348054/jobs/9758026382#step:5:8430 Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61116">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61116">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

git HEAD

### Custom Code

No

### OS Platform and Distribution

Ubuntu 20.04

### Mobile device

n/a

### Python version

3.9.16

### Bazel version

6.1.0

### GCC/Compiler version

10.2.1

### CUDA/cuDNN version

n/a

### GPU model and memory

n/a

### Current Behaviour?

//tensorflow/python/data/kernel_tests:snapshot_test sometimes fails

x86 log
https://source.cloud.google.com/results/invocations/8b60bfba-b6b6-4503-aa43-62e8bbe1a094/log

AARCH64 log


### Standalone code to reproduce the issue

```shell
bazel --bazelrc=/usertools/cpu.bazelrc test --config=pycpp --config=build_event_export --remote_cache=https://storage.googleapis.com/tensorflow-devinfra-bazel-cache/norbe --google_default_credentials
```


### Relevant log output

```shell
ERROR: testWriteSnapshotDatasetSameFingerprintIncompleteRunRestart_test_mode_eager_tfapiversion_2 (__main__.SnapshotTest)
SnapshotTest.testWriteSnapshotDatasetSameFingerprintIncompleteRunRestart_test_mode_eager_tfapiversion_2
testWriteSnapshotDatasetSameFingerprintIncompleteRunRestart_test_mode_eager_tfapiversion_2(mode='eager', tf_api_version=2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/kernel_tests/snapshot_test.runfiles/org_tensorflow/tensorflow/python/data/kernel_tests/snapshot_test.py", line 63, in tearDown
    shutil.rmtree(self._snapshot_dir)
  File "/usr/lib/python3.9/shutil.py", line 734, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.9/shutil.py", line 673, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/usr/lib/python3.9/shutil.py", line 671, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: '5643068742232426178'

======================================================================
FAIL: testWriteSnapshotDatasetSameFingerprintIncompleteRunRestart_test_mode_eager_tfapiversion_2 (__main__.SnapshotTest)
SnapshotTest.testWriteSnapshotDatasetSameFingerprintIncompleteRunRestart_test_mode_eager_tfapiversion_2
testWriteSnapshotDatasetSameFingerprintIncompleteRunRestart_test_mode_eager_tfapiversion_2(mode='eager', tf_api_version=2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/kernel_tests/snapshot_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test
    return test_method(self, **testcase_params)
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/kernel_tests/snapshot_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated
    execute_test_method()
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/kernel_tests/snapshot_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method
    test_method(**kwargs_to_pass)
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/kernel_tests/snapshot_test.runfiles/org_tensorflow/tensorflow/python/data/kernel_tests/snapshot_test.py", line 318, in testWriteSnapshotDatasetSameFingerprintIncompleteRunRestart
    self.assertSnapshotDirectoryContains(
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/kernel_tests/snapshot_test.runfiles/org_tensorflow/tensorflow/python/data/kernel_tests/snapshot_test.py", line 108, in assertSnapshotDirectoryContains
    self.assertLen(run_dirlist, num_snapshot_shards_per_run)
AssertionError: ['00000000.shard', '00000001.shard', '00000002.shard', '00000003.shard', '00000004.shard', '00000005.shard', '00000006.shard', '00000007.shard', '00000008.shard', '00000009.shard', '00000010.shard', '00000011.shard', '00000012.shard', '00000013.shard', '00000014.shard', '00000015.shard', '00000016.shard', '00000017.shard', '00000018.shard', '00000019.shard', '00000020.shard', '00000021.shard', '00000022.shard', '00000023.shard', '00000024.shard', '00000025.shard', '00000026.shard', '00000027.shard', '00000028.shard', '00000029.shard', '00000030.shard', '00000031.shard', '00000032.shard', '00000033.shard', '00000034.shard', '00000035.shard', '00000036.shard', '00000037.shard', '00000038.shard', '00000039.shard', '00000040.shard', '00000041.shard', '00000042.shard', '00000043.shard', '00000044.shard', '00000045.shard', '00000046.shard', '00000047.shard', '00000048.shard', '00000049.shard', '00000050.shard', '00000051.shard', '00000052.shard', '00000053.shard', '00000054.shard', '00000055.shard', '00000056.shard', '00000057.shard', '00000058.shard', '00000059.shard', '00000060.shard', '00000061.shard', '00000062.shard', '00000063.shard', '00000064.shard', '00000065.shard', '00000066.shard', '00000067.shard', '00000068.shard', '00000069.shard', '00000070.shard', '00000071.shard', '00000072.shard', '00000073.shard', '00000074.shard', '00000075.shard', '00000076.shard', '00000077.shard', '00000078.shard', '00000079.shard', '00000080.shard', '00000081.shard', '00000082.shard', '00000083.shard', '00000084.shard', '00000085.shard', '00000086.shard', '00000087.shard', '00000088.shard', '00000089.shard', '00000090.shard', '00000091.shard', '00000092.shard', '00000093.shard', '00000094.shard', '00000095.shard', '00000096.shard', '00000099.shard', '00000101.shard', '00000103.shard', '00000104.shard', '00000105.shard', '00000108.shard', '00000110.shard', '00000115.shard', '00000116.shard', '00000121.shard'] has length of 107, expected 128.

----------------------------------------------------------------------
Ran 20 tests in 5.563s
```
</details>