Based on FP16, the network training of resnet50 was carried out, and the existence accuracy randomly converged to 0.76
The training code used comes from github(https://github.com/tensorflow/models/tree/v2.11.0) From the output you have provided, it does not look like a random convergence.

Also, as per the ResNet training with vanilla settings with a batch size of 4096 and 90 epochs, the benchmark result is what you are getting.
https://github.com/tensorflow/models/tree/master/official/vision#resnet-models-trained-with-vanilla-settings I have repeated the training many times, sometimes it can converge to 0.76, sometimes it can only achieve 0.7 accuracy, I want to see if there are any good methods or tools to help me debug? Thanks. It is working as expected, you can not always achieve the same amount of accuracy for each run.
The reason you are getting `0.7` sometimes is because of the learning rate reduction stepwise set in the boundary limit.
After certain point, training will become saturated. 
```
    learning_rate:
      type: 'stepwise'
      stepwise:
        boundaries: [18750, 37500, 50000]
        values: [0.8, 0.08, 0.008, 0.0008]
```
Feel free to change the configuration file which you have specified during training, for example [here](https://github.com/tensorflow/models/blob/master/official/vision/configs/experiments/image_classification/imagenet_resnet50_gpu.yaml).  However, the settings of my training are the same every time, and the boundary is also the same value. I have trained for 90 epochs, so I feel that the accuracy of network training random convergence to 0.76 is not normal, I may now suspect that the calculation error of some kernels is a bit large。Now the network is built based on tf.keras.model and run with model.fit. Even if I use callback  function or print directly, I can’t grab the output of each layer, so I would like to ask if you have any suggestions or methods? Can help me grab the forward and backward calculation output of each layer, or are there other analysis methods? Thank you.  Could you please share the reproducible code to replicate the behavior and also sample data if you can share would be helpful to assist you better. Thanks! I have seted the seed by tf.random.set_seed(seed)、np.random.seed(seed）、random.seed(seed)、os.environ['PYTHONHASHSEED'] = str(seed)、os.environ['TF_DETERMINISTIC_OPS'] = '1'，and cancel the random of the imageimagenet_preprocessing.py, so i can get the reproduce behavior, Now the network is built based on tf.keras.model and run with model.fit. Even if I use callback function or print directly, I can’t grab the output of each layer, so I would like to ask if you have any suggestions or methods? Can help me grab the forward and backward calculation output of each layer, or are there other analysis methods? Thank you.
![3e560010163425fdac6088aaec7a9c0](https://github.com/tensorflow/tensorflow/assets/30514703/ac194191-dc6b-4506-95a3-17b7482f8a65)
 Hi, This issue seems to be specific to models, could you please close this issue and open a new issue in the repo https://github.com/tensorflow/models/issues I have already  take issue for there, but  have no anwser.  Does TensorFlow have any precision detection tools? It can be plugged directly into network training, similar to PyTorch's Torch GradCheck. Have any questions to plugged directly into network training, similar to PyTorch's Torch GradCheck.
### Issue type

Support

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

tf2.11

### Custom code

Yes

### OS platform and distribution

centos7.6

### Mobile device

_No response_

### Python version

3.7、3.8、3,。9

### Bazel version

5.3.0

### GCC/compiler version

9.4.0

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

Using the inagenet dataset, on the resnet50 network, based on FP16 training with the model-2.11.0 code,  the network has random convergence, is this phenomenon known, is there any solution at present?  Is there any good way to debug this problem? Thanks.

### Standalone code to reproduce the issue

```shell
python3  /public/home/TF_test/rocm5.4/tf2.11/models-master_fp16/official/benchmark/models/resnet_imagenet_main.py      --num_gpus=4     --batch_size=512  --train_epochs=90  --use_synthetic_data=false    --data_dir=/public/software/apps/DeepLearning/Data/ImageNet-tensorflow/  --enable_checkpoint_and_export  --dtype=fp16   --model_dir=./Checkpoint/
```


### Relevant log output

```shell
I0717 10:42:45.093785 47025319389696 controller.py:291]  eval | step:  10009 | eval time:   74.7 sec | output: {'test_accuracy': 0.08882, 'test_loss': 1.2578498}
I0717 11:19:43.333361 47025319389696 controller.py:291]  eval | step:  20018 | eval time:   47.9 sec | output: {'test_accuracy': 0.18752, 'test_loss': 1.0495228}
I0717 11:56:49.300491 47025319389696 controller.py:291]  eval | step:  30027 | eval time:   48.0 sec | output: {'test_accuracy': 0.2584, 'test_loss': 0.9102226}
I0717 12:33:43.809209 47025319389696 controller.py:291]  eval | step:  40036 | eval time:   47.3 sec | output: {'test_accuracy': 0.33168, 'test_loss': 0.77455497}
I0717 13:10:37.399015 47025319389696 controller.py:291]  eval | step:  50045 | eval time:   48.6 sec | output: {'test_accuracy': 0.32974, 'test_loss': 0.7866249}
I0717 13:47:31.637742 47025319389696 controller.py:291]  eval | step:  60054 | eval time:   48.0 sec | output: {'test_accuracy': 0.32832, 'test_loss': 0.7928696}
I0717 14:24:25.244748 47025319389696 controller.py:291]  eval | step:  70063 | eval time:   47.4 sec | output: {'test_accuracy': 0.36952, 'test_loss': 0.7357325}
I0717 15:01:26.831945 47025319389696 controller.py:291]  eval | step:  80072 | eval time:   49.6 sec | output: {'test_accuracy': 0.33716, 'test_loss': 0.7839137}
I0717 15:39:10.716763 47025319389696 controller.py:291]  eval | step:  90081 | eval time:   51.2 sec | output: {'test_accuracy': 0.32314, 'test_loss': 0.82485163}
I0717 16:17:14.804115 47025319389696 controller.py:291]  eval | step:  100090 | eval time:   49.1 sec | output: {'test_accuracy': 0.36102, 'test_loss': 0.7599265}
I0717 16:54:38.272225 47025319389696 controller.py:291]  eval | step:  110099 | eval time:   49.5 sec | output: {'test_accuracy': 0.42732, 'test_loss': 0.6477533}
I0717 17:32:14.305328 47025319389696 controller.py:291]  eval | step:  120108 | eval time:   49.1 sec | output: {'test_accuracy': 0.3769, 'test_loss': 0.745124}
I0717 18:19:19.995545 47025319389696 controller.py:291]  eval | step:  130117 | eval time:  196.2 sec | output: {'test_accuracy': 0.43222, 'test_loss': 0.63742775}
I0717 19:01:38.542571 47025319389696 controller.py:291]  eval | step:  140126 | eval time:   49.1 sec | output: {'test_accuracy': 0.43536, 'test_loss': 0.63410264}
I0717 19:39:52.303068 47025319389696 controller.py:291]  eval | step:  150135 | eval time:   46.7 sec | output: {'test_accuracy': 0.43782, 'test_loss': 0.6309778}
I0717 20:17:52.557711 47025319389696 controller.py:291]  eval | step:  160144 | eval time:   49.5 sec | output: {'test_accuracy': 0.44036, 'test_loss': 0.6212317}
I0717 20:55:54.236690 47025319389696 controller.py:291]  eval | step:  170153 | eval time:   47.7 sec | output: {'test_accuracy': 0.4211, 'test_loss': 0.6638591}
I0717 21:33:46.721763 47025319389696 controller.py:291]  eval | step:  180162 | eval time:   48.7 sec | output: {'test_accuracy': 0.41044, 'test_loss': 0.6806779}
I0717 22:11:56.432586 47025319389696 controller.py:291]  eval | step:  190171 | eval time:   48.9 sec | output: {'test_accuracy': 0.4227, 'test_loss': 0.6592407}
I0717 22:49:48.895009 47025319389696 controller.py:291]  eval | step:  200180 | eval time:   47.1 sec | output: {'test_accuracy': 0.41614, 'test_loss': 0.6689654}
I0717 23:27:56.641305 47025319389696 controller.py:291]  eval | step:  210189 | eval time:   49.0 sec | output: {'test_accuracy': 0.43086, 'test_loss': 0.6471556}
I0718 00:06:08.226011 47025319389696 controller.py:291]  eval | step:  220198 | eval time:   49.0 sec | output: {'test_accuracy': 0.43288, 'test_loss': 0.651372}
I0718 00:43:44.846733 47025319389696 controller.py:291]  eval | step:  230207 | eval time:   48.6 sec | output: {'test_accuracy': 0.44876, 'test_loss': 0.61514467}
I0718 01:21:01.860804 47025319389696 controller.py:291]  eval | step:  240216 | eval time:   47.8 sec | output: {'test_accuracy': 0.39994, 'test_loss': 0.6896438}
I0718 01:58:26.185002 47025319389696 controller.py:291]  eval | step:  250225 | eval time:   49.3 sec | output: {'test_accuracy': 0.43974, 'test_loss': 0.63887787}
I0718 02:35:51.395891 47025319389696 controller.py:291]  eval | step:  260234 | eval time:   48.7 sec | output: {'test_accuracy': 0.42148, 'test_loss': 0.65387714}
I0718 03:12:58.957236 47025319389696 controller.py:291]  eval | step:  270243 | eval time:   48.0 sec | output: {'test_accuracy': 0.42988, 'test_loss': 0.64366835}
I0718 03:50:05.619722 47025319389696 controller.py:291]  eval | step:  280252 | eval time:   49.1 sec | output: {'test_accuracy': 0.4292, 'test_loss': 0.64544094}
I0718 04:27:19.267962 47025319389696 controller.py:291]  eval | step:  290261 | eval time:   47.7 sec | output: {'test_accuracy': 0.43688, 'test_loss': 0.633886}
I0718 05:04:16.388976 47025319389696 controller.py:291]  eval | step:  300270 | eval time:   47.0 sec | output: {'test_accuracy': 0.4378, 'test_loss': 0.6427796}
I0718 05:41:28.633920 47025319389696 controller.py:291]  eval | step:  310279 | eval time:   47.1 sec | output: {'test_accuracy': 0.60604, 'test_loss': 0.41581097}
I0718 06:18:23.006313 47025319389696 controller.py:291]  eval | step:  320288 | eval time:   47.6 sec | output: {'test_accuracy': 0.60958, 'test_loss': 0.41247183}
I0718 06:55:19.940962 47025319389696 controller.py:291]  eval | step:  330297 | eval time:   47.5 sec | output: {'test_accuracy': 0.62418, 'test_loss': 0.3925642}
I0718 07:32:11.406366 47025319389696 controller.py:291]  eval | step:  340306 | eval time:   49.3 sec | output: {'test_accuracy': 0.60878, 'test_loss': 0.41140762}
I0718 08:09:13.098378 47025319389696 controller.py:291]  eval | step:  350315 | eval time:   46.3 sec | output: {'test_accuracy': 0.61416, 'test_loss': 0.40241298}
I0718 08:46:11.470759 47025319389696 controller.py:291]  eval | step:  360324 | eval time:   48.7 sec | output: {'test_accuracy': 0.61666, 'test_loss': 0.40103617}
I0718 09:23:19.292029 47025319389696 controller.py:291]  eval | step:  370333 | eval time:   47.6 sec | output: {'test_accuracy': 0.64238, 'test_loss': 0.3717267}
I0718 09:59:58.652599 47025319389696 controller.py:291]  eval | step:  380342 | eval time:   46.9 sec | output: {'test_accuracy': 0.63504, 'test_loss': 0.37905875}
I0718 10:36:30.269412 47025319389696 controller.py:291]  eval | step:  390351 | eval time:   46.7 sec | output: {'test_accuracy': 0.62588, 'test_loss': 0.38983408}
I0718 11:12:59.710509 47025319389696 controller.py:291]  eval | step:  400360 | eval time:   47.5 sec | output: {'test_accuracy': 0.62752, 'test_loss': 0.39042756}
I0718 11:49:34.702973 47025319389696 controller.py:291]  eval | step:  410369 | eval time:   47.6 sec | output: {'test_accuracy': 0.63086, 'test_loss': 0.38619643}
I0718 12:26:10.658050 47025319389696 controller.py:291]  eval | step:  420378 | eval time:   46.4 sec | output: {'test_accuracy': 0.61492, 'test_loss': 0.40776753}
I0718 13:02:54.373109 47025319389696 controller.py:291]  eval | step:  430387 | eval time:   47.1 sec | output: {'test_accuracy': 0.61482, 'test_loss': 0.40698445}
I0718 13:39:56.424064 47025319389696 controller.py:291]  eval | step:  440396 | eval time:   46.6 sec | output: {'test_accuracy': 0.61064, 'test_loss': 0.40885204}
I0718 14:16:56.972168 47025319389696 controller.py:291]  eval | step:  450405 | eval time:   47.2 sec | output: {'test_accuracy': 0.6101, 'test_loss': 0.41232613}
I0718 14:53:49.710101 47025319389696 controller.py:291]  eval | step:  460414 | eval time:   50.2 sec | output: {'test_accuracy': 0.62022, 'test_loss': 0.39717302}
I0718 15:31:00.055065 47025319389696 controller.py:291]  eval | step:  470423 | eval time:   47.8 sec | output: {'test_accuracy': 0.63738, 'test_loss': 0.3755849}
I0718 16:26:43.115331 47025319389696 controller.py:291]  eval | step:  480432 | eval time:   48.6 sec | output: {'test_accuracy': 0.63156, 'test_loss': 0.38420147}
I0718 17:14:57.232034 47025319389696 controller.py:291]  eval | step:  490441 | eval time:   47.9 sec | output: {'test_accuracy': 0.61762, 'test_loss': 0.40265194}
I0718 18:08:04.532505 47025319389696 controller.py:291]  eval | step:  500450 | eval time:   47.4 sec | output: {'test_accuracy': 0.63422, 'test_loss': 0.3809521}
I0718 18:45:18.862020 47025319389696 controller.py:291]  eval | step:  510459 | eval time:   50.4 sec | output: {'test_accuracy': 0.62098, 'test_loss': 0.39487627}
I0718 19:22:18.786072 47025319389696 controller.py:291]  eval | step:  520468 | eval time:   47.2 sec | output: {'test_accuracy': 0.61512, 'test_loss': 0.40484685}
I0718 19:59:17.472421 47025319389696 controller.py:291]  eval | step:  530477 | eval time:   47.3 sec | output: {'test_accuracy': 0.63776, 'test_loss': 0.37440324}
I0718 20:46:59.059379 47025319389696 controller.py:291]  eval | step:  540486 | eval time:  203.9 sec | output: {'test_accuracy': 0.62722, 'test_loss': 0.39110208}
I0718 21:45:11.991321 47025319389696 controller.py:291]  eval | step:  550495 | eval time:   47.9 sec | output: {'test_accuracy': 0.63562, 'test_loss': 0.38335004}
I0718 22:22:17.513960 47025319389696 controller.py:291]  eval | step:  560504 | eval time:   46.5 sec | output: {'test_accuracy': 0.61826, 'test_loss': 0.40100682}
I0718 22:59:05.393987 47025319389696 controller.py:291]  eval | step:  570513 | eval time:   46.1 sec | output: {'test_accuracy': 0.63496, 'test_loss': 0.3817627}
I0718 23:36:01.059101 47025319389696 controller.py:291]  eval | step:  580522 | eval time:   47.2 sec | output: {'test_accuracy': 0.61354, 'test_loss': 0.40860355}
I0719 00:12:57.767053 47025319389696 controller.py:291]  eval | step:  590531 | eval time:   48.7 sec | output: {'test_accuracy': 0.62766, 'test_loss': 0.3902482}
I0719 00:50:08.519172 47025319389696 controller.py:291]  eval | step:  600540 | eval time:   47.3 sec | output: {'test_accuracy': 0.62992, 'test_loss': 0.38722458}
I0719 01:27:14.241234 47025319389696 controller.py:291]  eval | step:  610549 | eval time:   49.5 sec | output: {'test_accuracy': 0.6785, 'test_loss': 0.3319877}
I0719 02:04:12.080565 47025319389696 controller.py:291]  eval | step:  620558 | eval time:   47.3 sec | output: {'test_accuracy': 0.68068, 'test_loss': 0.3297141}
I0719 02:41:31.931695 47025319389696 controller.py:291]  eval | step:  630567 | eval time:   47.5 sec | output: {'test_accuracy': 0.67544, 'test_loss': 0.33620217}
I0719 03:18:53.615640 47025319389696 controller.py:291]  eval | step:  640576 | eval time:   47.8 sec | output: {'test_accuracy': 0.67902, 'test_loss': 0.32960042}
I0719 03:56:03.865268 47025319389696 controller.py:291]  eval | step:  650585 | eval time:   47.2 sec | output: {'test_accuracy': 0.68312, 'test_loss': 0.32697994}
I0719 04:33:13.600398 47025319389696 controller.py:291]  eval | step:  660594 | eval time:   49.1 sec | output: {'test_accuracy': 0.68692, 'test_loss': 0.3218242}
I0719 05:10:26.761108 47025319389696 controller.py:291]  eval | step:  670603 | eval time:   47.5 sec | output: {'test_accuracy': 0.68098, 'test_loss': 0.33020604}
I0719 05:47:24.072698 47025319389696 controller.py:291]  eval | step:  680612 | eval time:   47.8 sec | output: {'test_accuracy': 0.68784, 'test_loss': 0.32333767}
I0719 06:24:23.891359 47025319389696 controller.py:291]  eval | step:  690621 | eval time:   47.1 sec | output: {'test_accuracy': 0.68598, 'test_loss': 0.32332215}
I0719 07:01:34.794637 47025319389696 controller.py:291]  eval | step:  700630 | eval time:   48.9 sec | output: {'test_accuracy': 0.68566, 'test_loss': 0.3250208}
I0719 07:38:47.115363 47025319389696 controller.py:291]  eval | step:  710639 | eval time:   47.2 sec | output: {'test_accuracy': 0.68696, 'test_loss': 0.32350788}
I0719 08:15:57.504423 47025319389696 controller.py:291]  eval | step:  720648 | eval time:   52.0 sec | output: {'test_accuracy': 0.68832, 'test_loss': 0.3217886}
I0719 08:53:14.727922 47025319389696 controller.py:291]  eval | step:  730657 | eval time:   49.9 sec | output: {'test_accuracy': 0.6852, 'test_loss': 0.32565758}
I0719 09:30:20.804430 47025319389696 controller.py:291]  eval | step:  740666 | eval time:   49.2 sec | output: {'test_accuracy': 0.68326, 'test_loss': 0.33065847}
I0719 10:07:20.642869 47025319389696 controller.py:291]  eval | step:  750675 | eval time:   48.0 sec | output: {'test_accuracy': 0.68372, 'test_loss': 0.32719776}
I0719 10:46:23.875730 47025319389696 controller.py:291]  eval | step:  760684 | eval time:   47.9 sec | output: {'test_accuracy': 0.68192, 'test_loss': 0.32962936}
I0719 11:23:17.199512 47025319389696 controller.py:291]  eval | step:  770693 | eval time:   47.2 sec | output: {'test_accuracy': 0.68262, 'test_loss': 0.32934195}
I0719 12:00:04.367842 47025319389696 controller.py:291]  eval | step:  780702 | eval time:   46.9 sec | output: {'test_accuracy': 0.67706, 'test_loss': 0.3367395}
I0719 12:37:04.218576 47025319389696 controller.py:291]  eval | step:  790711 | eval time:   49.9 sec | output: {'test_accuracy': 0.68592, 'test_loss': 0.32515442}
I0719 13:14:02.169136 47025319389696 controller.py:291]  eval | step:  800720 | eval time:   49.5 sec | output: {'test_accuracy': 0.6774, 'test_loss': 0.33403492}
I0719 13:51:13.950588 47025319389696 controller.py:291]  eval | step:  810729 | eval time:   48.3 sec | output: {'test_accuracy': 0.68796, 'test_loss': 0.32154444}
I0719 14:28:19.452734 47025319389696 controller.py:291]  eval | step:  820738 | eval time:   47.6 sec | output: {'test_accuracy': 0.68378, 'test_loss': 0.32751894}
I0719 15:05:49.973363 47025319389696 controller.py:291]  eval | step:  830747 | eval time:   47.5 sec | output: {'test_accuracy': 0.6856, 'test_loss': 0.3268724}
I0719 15:42:51.213348 47025319389696 controller.py:291]  eval | step:  840756 | eval time:   46.4 sec | output: {'test_accuracy': 0.68698, 'test_loss': 0.32407942}
I0719 16:19:54.159197 47025319389696 controller.py:291]  eval | step:  850765 | eval time:   48.8 sec | output: {'test_accuracy': 0.6875, 'test_loss': 0.32387343}
I0719 16:57:06.514332 47025319389696 controller.py:291]  eval | step:  860774 | eval time:   48.9 sec | output: {'test_accuracy': 0.6877, 'test_loss': 0.32420784}
I0719 17:34:12.212647 47025319389696 controller.py:291]  eval | step:  870783 | eval time:   48.5 sec | output: {'test_accuracy': 0.688, 'test_loss': 0.3242049}
I0719 18:11:14.854687 47025319389696 controller.py:291]  eval | step:  880792 | eval time:   48.6 sec | output: {'test_accuracy': 0.6882, 'test_loss': 0.324394}
I0719 18:48:31.401537 47025319389696 controller.py:291]  eval | step:  890801 | eval time:   49.7 sec | output: {'test_accuracy': 0.68516, 'test_loss': 0.32722312}
I0719 19:25:37.033689 47025319389696 controller.py:291]  eval | step:  900810 | eval time:   48.7 sec | output: {'test_accuracy': 0.68694, 'test_loss': 0.32629344}
 eval | step:  10009 | eval time:   74.7 sec | output: {'test_accuracy': 0.08882, 'test_loss': 1.2578498}
 eval | step:  20018 | eval time:   47.9 sec | output: {'test_accuracy': 0.18752, 'test_loss': 1.0495228}
 eval | step:  30027 | eval time:   48.0 sec | output: {'test_accuracy': 0.2584, 'test_loss': 0.9102226}
 eval | step:  40036 | eval time:   47.3 sec | output: {'test_accuracy': 0.33168, 'test_loss': 0.77455497}
 eval | step:  50045 | eval time:   48.6 sec | output: {'test_accuracy': 0.32974, 'test_loss': 0.7866249}
 eval | step:  60054 | eval time:   48.0 sec | output: {'test_accuracy': 0.32832, 'test_loss': 0.7928696}
 eval | step:  70063 | eval time:   47.4 sec | output: {'test_accuracy': 0.36952, 'test_loss': 0.7357325}
 eval | step:  80072 | eval time:   49.6 sec | output: {'test_accuracy': 0.33716, 'test_loss': 0.7839137}
 eval | step:  90081 | eval time:   51.2 sec | output: {'test_accuracy': 0.32314, 'test_loss': 0.82485163}
 eval | step:  100090 | eval time:   49.1 sec | output: {'test_accuracy': 0.36102, 'test_loss': 0.7599265}
 eval | step:  110099 | eval time:   49.5 sec | output: {'test_accuracy': 0.42732, 'test_loss': 0.6477533}
 eval | step:  120108 | eval time:   49.1 sec | output: {'test_accuracy': 0.3769, 'test_loss': 0.745124}
 eval | step:  130117 | eval time:  196.2 sec | output: {'test_accuracy': 0.43222, 'test_loss': 0.63742775}
 eval | step:  140126 | eval time:   49.1 sec | output: {'test_accuracy': 0.43536, 'test_loss': 0.63410264}
 eval | step:  150135 | eval time:   46.7 sec | output: {'test_accuracy': 0.43782, 'test_loss': 0.6309778}
 eval | step:  160144 | eval time:   49.5 sec | output: {'test_accuracy': 0.44036, 'test_loss': 0.6212317}
 eval | step:  170153 | eval time:   47.7 sec | output: {'test_accuracy': 0.4211, 'test_loss': 0.6638591}
 eval | step:  180162 | eval time:   48.7 sec | output: {'test_accuracy': 0.41044, 'test_loss': 0.6806779}
 eval | step:  190171 | eval time:   48.9 sec | output: {'test_accuracy': 0.4227, 'test_loss': 0.6592407}
 eval | step:  200180 | eval time:   47.1 sec | output: {'test_accuracy': 0.41614, 'test_loss': 0.6689654}
 eval | step:  210189 | eval time:   49.0 sec | output: {'test_accuracy': 0.43086, 'test_loss': 0.6471556}
 eval | step:  220198 | eval time:   49.0 sec | output: {'test_accuracy': 0.43288, 'test_loss': 0.651372}
 eval | step:  230207 | eval time:   48.6 sec | output: {'test_accuracy': 0.44876, 'test_loss': 0.61514467}
 eval | step:  240216 | eval time:   47.8 sec | output: {'test_accuracy': 0.39994, 'test_loss': 0.6896438}
 eval | step:  250225 | eval time:   49.3 sec | output: {'test_accuracy': 0.43974, 'test_loss': 0.63887787}
 eval | step:  260234 | eval time:   48.7 sec | output: {'test_accuracy': 0.42148, 'test_loss': 0.65387714}
 eval | step:  270243 | eval time:   48.0 sec | output: {'test_accuracy': 0.42988, 'test_loss': 0.64366835}
 eval | step:  280252 | eval time:   49.1 sec | output: {'test_accuracy': 0.4292, 'test_loss': 0.64544094}
 eval | step:  290261 | eval time:   47.7 sec | output: {'test_accuracy': 0.43688, 'test_loss': 0.633886}
 eval | step:  300270 | eval time:   47.0 sec | output: {'test_accuracy': 0.4378, 'test_loss': 0.6427796}
 eval | step:  310279 | eval time:   47.1 sec | output: {'test_accuracy': 0.60604, 'test_loss': 0.41581097}
 eval | step:  320288 | eval time:   47.6 sec | output: {'test_accuracy': 0.60958, 'test_loss': 0.41247183}
 eval | step:  330297 | eval time:   47.5 sec | output: {'test_accuracy': 0.62418, 'test_loss': 0.3925642}
 eval | step:  340306 | eval time:   49.3 sec | output: {'test_accuracy': 0.60878, 'test_loss': 0.41140762}
 eval | step:  350315 | eval time:   46.3 sec | output: {'test_accuracy': 0.61416, 'test_loss': 0.40241298}
 eval | step:  360324 | eval time:   48.7 sec | output: {'test_accuracy': 0.61666, 'test_loss': 0.40103617}
 eval | step:  370333 | eval time:   47.6 sec | output: {'test_accuracy': 0.64238, 'test_loss': 0.3717267}
 eval | step:  380342 | eval time:   46.9 sec | output: {'test_accuracy': 0.63504, 'test_loss': 0.37905875}
 eval | step:  390351 | eval time:   46.7 sec | output: {'test_accuracy': 0.62588, 'test_loss': 0.38983408}
 eval | step:  400360 | eval time:   47.5 sec | output: {'test_accuracy': 0.62752, 'test_loss': 0.39042756}
 eval | step:  410369 | eval time:   47.6 sec | output: {'test_accuracy': 0.63086, 'test_loss': 0.38619643}
 eval | step:  420378 | eval time:   46.4 sec | output: {'test_accuracy': 0.61492, 'test_loss': 0.40776753}
 eval | step:  430387 | eval time:   47.1 sec | output: {'test_accuracy': 0.61482, 'test_loss': 0.40698445}
 eval | step:  440396 | eval time:   46.6 sec | output: {'test_accuracy': 0.61064, 'test_loss': 0.40885204}
 eval | step:  450405 | eval time:   47.2 sec | output: {'test_accuracy': 0.6101, 'test_loss': 0.41232613}
 eval | step:  460414 | eval time:   50.2 sec | output: {'test_accuracy': 0.62022, 'test_loss': 0.39717302}
 eval | step:  470423 | eval time:   47.8 sec | output: {'test_accuracy': 0.63738, 'test_loss': 0.3755849}
 eval | step:  480432 | eval time:   48.6 sec | output: {'test_accuracy': 0.63156, 'test_loss': 0.38420147}
 eval | step:  490441 | eval time:   47.9 sec | output: {'test_accuracy': 0.61762, 'test_loss': 0.40265194}
 eval | step:  500450 | eval time:   47.4 sec | output: {'test_accuracy': 0.63422, 'test_loss': 0.3809521}
 eval | step:  510459 | eval time:   50.4 sec | output: {'test_accuracy': 0.62098, 'test_loss': 0.39487627}
 eval | step:  520468 | eval time:   47.2 sec | output: {'test_accuracy': 0.61512, 'test_loss': 0.40484685}
 eval | step:  530477 | eval time:   47.3 sec | output: {'test_accuracy': 0.63776, 'test_loss': 0.37440324}
 eval | step:  540486 | eval time:  203.9 sec | output: {'test_accuracy': 0.62722, 'test_loss': 0.39110208}
 eval | step:  550495 | eval time:   47.9 sec | output: {'test_accuracy': 0.63562, 'test_loss': 0.38335004}
 eval | step:  560504 | eval time:   46.5 sec | output: {'test_accuracy': 0.61826, 'test_loss': 0.40100682}
 eval | step:  570513 | eval time:   46.1 sec | output: {'test_accuracy': 0.63496, 'test_loss': 0.3817627}
 eval | step:  580522 | eval time:   47.2 sec | output: {'test_accuracy': 0.61354, 'test_loss': 0.40860355}
 eval | step:  590531 | eval time:   48.7 sec | output: {'test_accuracy': 0.62766, 'test_loss': 0.3902482}
 eval | step:  600540 | eval time:   47.3 sec | output: {'test_accuracy': 0.62992, 'test_loss': 0.38722458}
 eval | step:  610549 | eval time:   49.5 sec | output: {'test_accuracy': 0.6785, 'test_loss': 0.3319877}
 eval | step:  620558 | eval time:   47.3 sec | output: {'test_accuracy': 0.68068, 'test_loss': 0.3297141}
 eval | step:  630567 | eval time:   47.5 sec | output: {'test_accuracy': 0.67544, 'test_loss': 0.33620217}
 eval | step:  640576 | eval time:   47.8 sec | output: {'test_accuracy': 0.67902, 'test_loss': 0.32960042}
 eval | step:  650585 | eval time:   47.2 sec | output: {'test_accuracy': 0.68312, 'test_loss': 0.32697994}
 eval | step:  660594 | eval time:   49.1 sec | output: {'test_accuracy': 0.68692, 'test_loss': 0.3218242}
 eval | step:  670603 | eval time:   47.5 sec | output: {'test_accuracy': 0.68098, 'test_loss': 0.33020604}
 eval | step:  680612 | eval time:   47.8 sec | output: {'test_accuracy': 0.68784, 'test_loss': 0.32333767}
 eval | step:  690621 | eval time:   47.1 sec | output: {'test_accuracy': 0.68598, 'test_loss': 0.32332215}
 eval | step:  700630 | eval time:   48.9 sec | output: {'test_accuracy': 0.68566, 'test_loss': 0.3250208}
 eval | step:  710639 | eval time:   47.2 sec | output: {'test_accuracy': 0.68696, 'test_loss': 0.32350788}
 eval | step:  720648 | eval time:   52.0 sec | output: {'test_accuracy': 0.68832, 'test_loss': 0.3217886}
 eval | step:  730657 | eval time:   49.9 sec | output: {'test_accuracy': 0.6852, 'test_loss': 0.32565758}
 eval | step:  740666 | eval time:   49.2 sec | output: {'test_accuracy': 0.68326, 'test_loss': 0.33065847}
 eval | step:  750675 | eval time:   48.0 sec | output: {'test_accuracy': 0.68372, 'test_loss': 0.32719776}
 eval | step:  760684 | eval time:   47.9 sec | output: {'test_accuracy': 0.68192, 'test_loss': 0.32962936}
 eval | step:  770693 | eval time:   47.2 sec | output: {'test_accuracy': 0.68262, 'test_loss': 0.32934195}
 eval | step:  780702 | eval time:   46.9 sec | output: {'test_accuracy': 0.67706, 'test_loss': 0.3367395}
 eval | step:  790711 | eval time:   49.9 sec | output: {'test_accuracy': 0.68592, 'test_loss': 0.32515442}
 eval | step:  800720 | eval time:   49.5 sec | output: {'test_accuracy': 0.6774, 'test_loss': 0.33403492}
 eval | step:  810729 | eval time:   48.3 sec | output: {'test_accuracy': 0.68796, 'test_loss': 0.32154444}
 eval | step:  820738 | eval time:   47.6 sec | output: {'test_accuracy': 0.68378, 'test_loss': 0.32751894}
 eval | step:  830747 | eval time:   47.5 sec | output: {'test_accuracy': 0.6856, 'test_loss': 0.3268724}
 eval | step:  840756 | eval time:   46.4 sec | output: {'test_accuracy': 0.68698, 'test_loss': 0.32407942}
 eval | step:  850765 | eval time:   48.8 sec | output: {'test_accuracy': 0.6875, 'test_loss': 0.32387343}
 eval | step:  860774 | eval time:   48.9 sec | output: {'test_accuracy': 0.6877, 'test_loss': 0.32420784}
 eval | step:  870783 | eval time:   48.5 sec | output: {'test_accuracy': 0.688, 'test_loss': 0.3242049}
 eval | step:  880792 | eval time:   48.6 sec | output: {'test_accuracy': 0.6882, 'test_loss': 0.324394}
 eval | step:  890801 | eval time:   49.7 sec | output: {'test_accuracy': 0.68516, 'test_loss': 0.32722312}
 eval | step:  900810 | eval time:   48.7 sec | output: {'test_accuracy': 0.68694, 'test_loss': 0.32629344}
```
