tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value num_blocks_2/multihead_attention/conv1d_1/kerne
Hi @vrunm ,

It seems you are using older TF versions.  tf.train.Saver is deprecated and use  tf.train.Checkpoint instead. Could you please submit a minimal code snippet to reproduce the issue with latest versions with the updated APIs ?

Please find the warning below from attached [gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/9d9984f2b1da1115f58e355d22d76e4a/61446.ipynb#scrollTo=4S4yrXCjRZZQ).

`WARNING:tensorflow:Saver is deprecated, please switch to tf.train.Checkpoint or tf.keras.Model.save_weights for training checkpoints. When executing eagerly variables do not necessarily have unique names, and so the variable.name-based lookups Saver performs are error-prone.`

Thanks !

 I am using tensorflow 1.14 so, this is not the problem in my case. It is depreciated only on tensorflow 2.x, where you have tested. Do you still offer the solution to older tensorflow versions? @vrunm ,

If it is of 1.x versions then currently we are not supporting. If possible please migrate to TF2.x versions and preferably latest versions.

Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61446">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61446">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

tf 2.8

### Custom code

Yes

### OS platform and distribution

Linux Ubuntu 20.04

### Mobile device

_No response_

### Python version

3.9

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

I have trained the [CARCA(Context and Attribute-Aware Sequential Recommendation via Cross-Attention)](https://github.com/ahmedrashed-ml/CARCA) on Video Games Dataset. I saved the session after 1 epoch and try to restore the session since all the Architecture was written using the concepts of the session. I saved the session using tf.train.Saver() .save() method.
```

session_saver = tf.train.Saver(save_relative_paths=True)
session_output_path = os.path.join(args.output_dir, "epochs_"+str(epoch))
if not os.path.isdir(session_output_path):
    os.makedirs(session_output_path)# make directory if not exists
session_saver.save(sess, session_output_path+"/carca_model")
print(f"[INFO]: Save the model after epochs: {epoch}")
```
then, I restored session using:
```
saver = tf.train.import_meta_graph(session_dir +"carca_model.meta")
saver.restore(sess, tf.train.latest_checkpoint(session_dir))
```
I have tried to perform prediction on new dataset using restored session sess, But I have encountered Attempting to use uninitialized value error. The full error is:
```

Traceback (most recent call last):
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value num_blocks_2/multihead_attention/conv1d_1/kernel_1
         [[{{node num_blocks_2/multihead_attention/conv1d_1/kernel_1/read}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "CARCA_train.py", line 1441, in <module>
    get_load_model_and_inference(dataset, usernum, itemnum, args, ItemFeatures, UserFeatures, CXTDict)
  File "CARCA_train.py", line 1317, in get_load_model_and_inference
    predictions = -model.predict(sess, np.ones(args.maxlen)*u, [seq], item_idx, [seqcxt], testitemscxt)
  File "CARCA_train.py", line 976, in predict
    {self.test_user: u, self.input_seq: seq, self.test_item: item_idx, self.is_training: False, self.seq_cxt:seqcxt, self.test_item_cxt:testitemcxt})
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value num_blocks_2/multihead_attention/conv1d_1/kernel_1
         [[node num_blocks_2/multihead_attention/conv1d_1/kernel_1/read (defined at CARCA_train.py:670) ]]

Original stack trace for 'num_blocks_2/multihead_attention/conv1d_1/kernel_1/read':
  File "CARCA_train.py", line 1441, in <module>
    get_load_model_and_inference(dataset, usernum, itemnum, args, ItemFeatures, UserFeatures, CXTDict)
  File "CARCA_train.py", line 1266, in get_load_model_and_inference
    model = Model(usernum, itemnum, args, ItemFeatures, UserFeatures, cxt_size = cxt_size ,use_res = True)
  File "CARCA_train.py", line 768, in __init__
    dropout_rate=args.dropout_rate, is_training=self.is_training)
  File "CARCA_train.py", line 670, in feedforward
    outputs = tf.layers.conv1d(**params)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/layers/convolutional.py", line 218, in conv1d
    return layer.apply(inputs)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1479, in apply
    return self.__call__(inputs, *args, **kwargs)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/layers/base.py", line 537, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 591, in __call__
    self._maybe_build(inputs)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1881, in _maybe_build
    self.build(input_shapes)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/keras/layers/convolutional.py", line 165, in build
    dtype=self.dtype)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/layers/base.py", line 450, in add_weight
    **kwargs)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 384, in add_weight
    aggregation=aggregation)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/training/tracking/base.py", line 663, in _add_variable_with_custom_getter
    **kwargs_for_getter)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1496, in get_variable
    aggregation=aggregation)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1239, in get_variable
    aggregation=aggregation)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 562, in get_variable
    aggregation=aggregation)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 514, in _true_getter
    aggregation=aggregation)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 929, in _get_single_variable
    aggregation=aggregation)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 259, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 220, in _variable_v1_call
    shape=shape)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 198, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 2511, in default_variable_creator
    shape=shape)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 263, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 1568, in __init__
    shape=shape)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 1755, in _init_from_args
    self._snapshot = array_ops.identity(self._variable, name="read")
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py", line 86, in identity
    ret = gen_array_ops.identity(input, name=name)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 4253, in identity
    "Identity", input=input, name=name)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/home/zakipoint/miniconda3/envs/sequential_recommendation_carca/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

I have also checked the variables in both the stored session and the session just after training. Both outputs seem similar. I was stuck on this issue for a few days and also tested saving the model using tf.saved_model.builder.SavedModelBuilder() and restoring the model using tf.saved_model.loader.load() but not solved the issue.Currently,I am using tensorflow 1.14
```

### Standalone code to reproduce the issue

```shell
session_saver = tf.train.Saver(save_relative_paths=True)
session_output_path = os.path.join(args.output_dir, "epochs_"+str(epoch))
if not os.path.isdir(session_output_path):
    os.makedirs(session_output_path)# make directory if not exists
session_saver.save(sess, session_output_path+"/carca_model")
print(f"[INFO]: Save the model after epochs: {epoch}")

saver = tf.train.import_meta_graph(session_dir +"carca_model.meta")
saver.restore(sess, tf.train.latest_checkpoint(session_dir))
```


### Relevant log output

_No response_