Avoid partially saved ckpt from preempted device (e.g. TPU)
@edwardyehuang,
Could you please provide the complete standalone code to reproduce the issue and it helps us to analyse the issue in an effective way. Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. > @edwardyehuang, Could you please provide the complete standalone code to reproduce the issue and it helps us to analyse the issue in an effective way. Thank you!

As I said above, as an individual user outside of Google, I cannot manually seize a TPU node, so it is impossible to provide an independent reproduction code.
By the way, I've only had this problem happen twice (I've experienced thousands of preemptions). @edwardyehuang,
Without the reproducible code, it would be difficult for us to debug the issue. In order to expedite the trouble-shooting process, could you please provide a minimal code snippet. Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61552">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61552">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

tf.2.11

### Custom code

Yes

### OS platform and distribution

_No response_

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

I accidentally discovered from a TPU preemption that my ckpt cannot be used to resume training (the preemption occurs during the process of saving ckpt). The error is that some weights cannot be matched. This phenomenon is happening for the first time and has never happened before.

### Standalone code to reproduce the issue

```shell
It is very hard to reproduce because it must be preemption while saving the ckpt.
```


### Relevant log output

_No response_