Error when trring a second training :  WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.iter
Hi @garywillcodeit ,

The error message seems generating from the destructor method from [here](https://github.com/tensorflow/tensorflow/blob/cb348a09747c484a987d3c4f5765c641e5b6489f/tensorflow/python/checkpoint/checkpoint.py#L234).

I am not sure how and why destructor method being called here. I request you to provide more context and minimal repro code for same.

Thanks. This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/65657">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/65657">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.13.0

### Custom code

Yes

### OS platform and distribution

MAC OS 13.5 Ventura

### Mobile device

_No response_

### Python version

3.11.9

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

Hi everyone !

I am trying to train a pre-trained TensorFlow 2 Detection Model Zoo. Everything works well after the first training, new checkpoints are generated and when I try detecting objects, it finds perfectly the objects in every images.

The problem is that when I start a new training (a second training), I get this message and the training stops :

**WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.iter**

Here is my tree structure:

_tensorflow
____models (TF API)
____workspace
________annotations
_____________label_map.pbtxt
_____________test.record
_____________train.record
________images
_____________train
_________________IMG1.jpg
_________________IMG1.xml
...
_________________IMG50.jpg
_________________IMG50.xml
_____________test
_________________IMG1.jpg
_________________IMG1.xml
...
_________________IMG20.jpg
_________________IMG20.xml
________models
_____________my_ssd_mobnet
_________________eval
_________________train
_________________checkpoint (file)
_________________ckpt-1.data-00000-of-00001
_________________ckpt-1.index
_________________ckpt-2.data-00000-of-00001
_________________ckpt-2.index
_________________ckpt-3.data-00000-of-00001
_________________ckpt-3.index
_________________pipeline.config
________pre-trained-model
_____________checkpoint (dir)
_________________checkpoint (file)
_________________ckpt-0.data-00000-of-00001
_________________ckpt-0.index
_____________saved_model
_________________variables
_____________________variables.data-00000-of-00001
_____________________variables.index
_________________saved_model.pb
_____________pipeline.config


 Here is my process:

1. I create the label map
2. I generate the TF records
3. I update pipeline.config: - num_classes (depending on labels' length); - fine_tune_checkpoint (I set the pre-trained-model checkpoint dir); - label_map_path (label_map.pbtxt path); - tf record for tain and eval
4. I run this code 
```
command = [
        "python",
        training_script, // This is the "model_main_tf2.py" path
        f"--model_dir={my_ssd_mobnet_path}",
        f"--pipeline_config_path={trained_pipeline}",
        "--num_train_steps=2000",
    ]

subprocess.run(command, text=True)
```

5. I evaluate the model
    
As I said before, everything works very well during the first training. The training goes until 2000 steps, the evaluation returns great data and when I try the trained model, it manages to find all objects that I trained it for.

But when I try to run a new (second) training, I get the warning message and the training stops.

I read [here](https://stackoverflow.com/questions/71929036/value-in-checkpoint-could-not-be-found-in-the-restored-object-root-optimizer) that changing the steps number works but not for me.

I also read that adding this line of code works but I don't see where to put it in mine  🤷🏼‍♂️

```
model.load_weights(MY_CHECKPOINT_FILE).expect_partial()
``` 

Things that work :

The only thing that works is to delete all the checkpoint that has been generated during the first training. But I finally understood that I also remove all data from the first training.

Has anyone experienced this problem and found a solution?


### Standalone code to reproduce the issue

```shell
command = [
        "python",
        training_script, // This is the "model_main_tf2.py" path
        f"--model_dir={my_ssd_mobnet_path}",
        f"--pipeline_config_path={trained_pipeline}",
        "--num_train_steps=2000",
    ]

subprocess.run(command, text=True)
```


### Relevant log output

```shell
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.iter
```
