Error about using TensorFlow1.14
@SupermanTai,
We see that you are using tf version 1.14, 1.x is not actively supported, please update to 2.x and let us know if you are facing the same issue. 

And also you have provided the code which is suitable for 1.x version. Could you please try to migrate your TensorFlow code from TensorFlow 1.x to TensorFlow 2 with the help of this official document.
https://www.tensorflow.org/guide/migrate

Also please try to install the tensorflow latest stable version by referring to the official build documentation.
https://www.tensorflow.org/install

Thank you! Tensorflow 1 is no longer supported, To ease the process of conversion you can install Tensorflow and Tensorflow-upgrade using the following Commands

`pip install tensorflow`

and

`pip install tensorflow-upgrade`

Run the upgrade script on your TensorFlow 1.x code:

`tensorflow-upgrade --infile your_script.py --outfile upgraded_script.py`

Replace **your_script.py** with the name to your TensorFlow 1.x script, this will remove all the depracated methods and replace them with new ones in the new script. This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61089">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61089">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Build/Install

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

tf 1.14.0

### Custom Code

Yes

### OS Platform and Distribution

Ubuntu18.04

### Mobile device

Ubuntu18.04

### Python version

3.7

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

cuda10.0 cudnn7.6.5

### GPU model and memory

24G

### Current Behaviour?

I use Ubuntu18.04 system, cuda10.0, cudnn7.6.4, graphics card is a30,24G; TensorFlow1.14.0 was used to make ner model, but an error occurred when running, and there was no error when using it before. It can be run on another device with no change in the code. I hope to get your reply and help.

### Standalone code to reproduce the issue

```shell
import tensorflow as tf
from tensorflow.contrib.crf import crf_log_likelihood
from tensorflow.contrib.layers.python.layers import initializers

from albert import modeling
import models.rnncell as rnn
from models.file_path import BaseConfig


class Config(BaseConfig):
    batch_size = 128
    epoch = 100
    print_per_batch = 100
    clip = 5
    dropout_keep_prob = 0.5
    lr = 0.0001
    optimizer = 'adam'
    zeros = False
    lower = True

    num_tags = None
    lstm_dim = 200
    max_seq_len = 256
    max_epoch = 100
    steps_check = 100


class AlbertBiLstmCrf(object):
    def __init__(self, config):
        self.config = config
        # 为模型添加占位符
        self.input_ids = tf.placeholder(dtype=tf.int32, shape=[None, None], name="input_ids")
        self.input_mask = tf.placeholder(dtype=tf.int32, shape=[None, None], name="input_mask")
        self.segment_ids = tf.placeholder(dtype=tf.int32, shape=[None, None], name="segment_ids")
        self.targets = tf.placeholder(dtype=tf.int32, shape=[None, None], name="Targets")

        self.dropout = tf.placeholder(dtype=tf.float32, name="Dropout")
        self.global_step = tf.Variable(0, trainable=False)
        self.best_dev_f1 = tf.Variable(0.0, trainable=False)

        self.initializer = initializers.xavier_initializer()
        self.albert_bilstm_crf()

    def albert_bilstm_crf(self):
        # parameter
        used = tf.sign(tf.abs(self.input_ids))
        length = tf.reduce_sum(used, reduction_indices=1)
        self.lengths = tf.cast(length, tf.int32)
        self.batch_size = tf.shape(self.input_ids)[0]
        self.num_steps = tf.shape(self.input_ids)[-1]

        # albert embedding
        embedding = self.bert_embedding()
        # dropout
        lstm_inputs = tf.nn.dropout(embedding, self.dropout)
        # bi-lstm layer
        lstm_outputs = self.biLSTM_layer(lstm_inputs, self.config.lstm_dim, self.lengths)
        # logits for tags
        self.logits = self.project_layer(lstm_outputs)
        # loss of the model
        self.loss = self.loss_layer(self.logits, self.lengths)

        # bert模型参数初始化的地方
        init_checkpoint = self.config.init_checkpoint
        # 获取模型中所有的训练参数。
        tvars = tf.trainable_variables()
        # 加载BERT模型
        (assignment_map, initialized_variable_names) = modeling.get_assignment_map_from_checkpoint(tvars, init_checkpoint)
        tf.train.init_from_checkpoint(init_checkpoint, assignment_map)
        # 打印加载模型的参数
        train_vars = []
        for var in tvars:
            init_string = ""
            if var.name in initialized_variable_names:
                init_string = ", *INIT_FROM_CKPT*"
            else:
                train_vars.append(var)
            # print("  name = %s, shape = %s%s", var.name, var.shape,init_string)

        optimizer = self.config.optimizer
        if optimizer == "adam":
            self.opt = tf.train.AdamOptimizer(self.config.lr)
        else:
            raise KeyError
        grads = tf.gradients(self.loss, train_vars)
        (grads, _) = tf.clip_by_global_norm(grads, clip_norm=1.0)

        self.train_op = self.opt.apply_gradients(zip(grads, train_vars), global_step=self.global_step)

    def bert_embedding(self):
        # load bert embedding
        albert_config = modeling.AlbertConfig.from_json_file(self.config.bert_config_path)  # 配置文件地址。
        model = modeling.AlbertModel(
            config=albert_config,
            is_training=True,
            input_ids=self.input_ids,
            input_mask=self.input_mask,
            token_type_ids=self.segment_ids,
            use_one_hot_embeddings=False)
        embedding = model.get_sequence_output()
        return embedding

    def biLSTM_layer(self, lstm_inputs, lstm_dim, lengths, name=None):

        #lstm_inputs: [batch_size, num_steps, emb_size] return：[batch_size, num_steps, 2*lstm_dim]

        with tf.variable_scope("char_BiLSTM" if not name else name):
            lstm_cell = {}
            for direction in ["forward", "backward"]:
                with tf.variable_scope(direction):
                    lstm_cell[direction] = rnn.CoupledInputForgetGateLSTMCell(
                        lstm_dim,
                        use_peepholes=True,
                        initializer=self.initializer,
                        state_is_tuple=True)
            outputs, final_states = tf.nn.bidirectional_dynamic_rnn(
                lstm_cell["forward"],
                lstm_cell["backward"],
                lstm_inputs,
                dtype=tf.float32,
                sequence_length=lengths)
        return tf.concat(outputs, axis=2)

    def project_layer(self, lstm_outputs, name=None):
        #lstm和logits之间的隐藏层
        #lstm_outputs: [batch_size, num_steps, emb_size] return: [batch_size, num_steps, num_tags]
        with tf.variable_scope("project" if not name else name):
            with tf.variable_scope("hidden"):
                W = tf.get_variable("W", shape=[self.config.lstm_dim * 2, self.config.lstm_dim],dtype=tf.float32, initializer=self.initializer)

                b = tf.get_variable("b", shape=[self.config.lstm_dim], dtype=tf.float32,initializer=tf.zeros_initializer())
                output = tf.reshape(lstm_outputs, shape=[-1, self.config.lstm_dim * 2])
                hidden = tf.tanh(tf.nn.xw_plus_b(output, W, b))

            # project to score of tags
            with tf.variable_scope("logits"):
                W = tf.get_variable("W", shape=[self.config.lstm_dim, self.config.num_tags],dtype=tf.float32, initializer=self.initializer)

                b = tf.get_variable("b", shape=[self.config.num_tags], dtype=tf.float32,initializer=tf.zeros_initializer())

                pred = tf.nn.xw_plus_b(hidden, W, b)

            return tf.reshape(pred, [-1, self.num_steps, self.config.num_tags])

    def loss_layer(self, project_logits, lengths, name=None):
        #计算crf损失
        with tf.variable_scope("crf_loss" if not name else name):
            small = -1000.0
            # pad logits for crf loss
            start_logits = tf.concat([small * tf.ones(shape=[self.batch_size, 1, self.config.num_tags]),tf.zeros(shape=[self.batch_size, 1, 1])],axis=-1)
            pad_logits = tf.cast(small * tf.ones([self.batch_size, self.num_steps, 1]), tf.float32)
            logits = tf.concat([project_logits, pad_logits], axis=-1)
            logits = tf.concat([start_logits, logits], axis=1)
            targets = tf.concat([tf.cast(self.config.num_tags * tf.ones([self.batch_size, 1]), tf.int32), self.targets], axis=-1)

            self.trans = tf.get_variable("transitions",shape=[self.config.num_tags + 1, self.config.num_tags + 1],initializer=self.initializer)
            #对数似然
            log_likelihood, self.trans = crf_log_likelihood(
                inputs=logits,
                tag_indices=targets,
                transition_params=self.trans,
                sequence_lengths=lengths + 1)
            return tf.reduce_mean(-log_likelihood)
```


### Relevant log output

```shell
2023-06-27 14:12:06.722268: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2023-06-27 14:12:06.728420: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2023-06-27 14:12:06.834462: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5647fd4400f0 executing computations on platform CUDA. Devices:
2023-06-27 14:12:06.834508: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): NVIDIA A30, Compute Capability 8.0
2023-06-27 14:12:06.838777: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2400000000 Hz
2023-06-27 14:12:06.839586: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5647fb24a440 executing computations on platform Host. Devices:
2023-06-27 14:12:06.839615: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2023-06-27 14:12:06.840801: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: NVIDIA A30 major: 8 minor: 0 memoryClockRate(GHz): 1.44
pciBusID: 0000:65:00.0
2023-06-27 14:12:06.841076: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2023-06-27 14:12:06.842475: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2023-06-27 14:12:06.843695: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2023-06-27 14:12:06.844007: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2023-06-27 14:12:06.845668: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2023-06-27 14:12:06.846956: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2023-06-27 14:12:06.850503: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2023-06-27 14:12:06.852002: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2023-06-27 14:12:06.852036: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2023-06-27 14:12:06.853078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-06-27 14:12:06.853091: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2023-06-27 14:12:06.853099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2023-06-27 14:12:06.854597: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 12129 MB memory) -> physical GPU (device: 0, name: NVIDIA A30, pci bus id: 0000:65:00.0, compute capability: 8.0)
2023-06-27 14:12:06,855 - /root/Tai/Project/ANER-规范性文件清单/log/train.log - INFO - Created model with fresh parameters.
2023-06-27 14:12:07.187507: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
2023-06-27 14:14:18,782 - /root/Tai/Project/ANER-规范性文件清单/log/train.log - INFO - start training
2023-06-27 14:14:20.096642: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2023-06-27 14:14:22.597785: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
Traceback (most recent call last):
  File "/root/anaconda3/envs/ner/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/root/anaconda3/envs/ner/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/root/anaconda3/envs/ner/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) Internal: Blas GEMM launch failed : a.shape=(16384, 2), b.shape=(2, 128), m=16384, n=128, k=2
	 [[{{node bert/embeddings/MatMul}}]]
	 [[Adam/update/_278]]
  (1) Internal: Blas GEMM launch failed : a.shape=(16384, 2), b.shape=(2, 128), m=16384, n=128, k=2
	 [[{{node bert/embeddings/MatMul}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 69, in <module>
    train(model, config, train_manager, dev_manager, id_to_tag)
  File "train.py", line 38, in train
    global_step, batch_loss, _ = sess.run([model.global_step, model.loss, model.train_op], feed_dict)
  File "/root/anaconda3/envs/ner/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/root/anaconda3/envs/ner/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/root/anaconda3/envs/ner/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/root/anaconda3/envs/ner/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) Internal: Blas GEMM launch failed : a.shape=(16384, 2), b.shape=(2, 128), m=16384, n=128, k=2
	 [[node bert/embeddings/MatMul (defined at /root/Tai/Project/ANER-规范性文件清单/albert/modeling.py:552) ]]
	 [[Adam/update/_278]]
  (1) Internal: Blas GEMM launch failed : a.shape=(16384, 2), b.shape=(2, 128), m=16384, n=128, k=2
	 [[node bert/embeddings/MatMul (defined at /root/Tai/Project/ANER-规范性文件清单/albert/modeling.py:552) ]]
0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node bert/embeddings/MatMul:
 bert/embeddings/token_type_embeddings/read (defined at /root/Tai/Project/ANER-规范性文件清单/albert/modeling.py:547)	
 bert/embeddings/one_hot (defined at /root/Tai/Project/ANER-规范性文件清单/albert/modeling.py:551)

Input Source operations connected to node bert/embeddings/MatMul:
 bert/embeddings/token_type_embeddings/read (defined at /root/Tai/Project/ANER-规范性文件清单/albert/modeling.py:547)	
 bert/embeddings/one_hot (defined at /root/Tai/Project/ANER-规范性文件清单/albert/modeling.py:551)

Original stack trace for 'bert/embeddings/MatMul':
  File "train.py", line 66, in <module>
    model = AlbertBiLstmCrf(config)
  File "/root/Tai/Project/ANER-规范性文件清单/models/modeling_BBC.py", line 42, in __init__
    self.albert_bilstm_crf()
  File "/root/Tai/Project/ANER-规范性文件清单/models/modeling_BBC.py", line 53, in albert_bilstm_crf
    embedding = self.bert_embedding()
  File "/root/Tai/Project/ANER-规范性文件清单/models/modeling_BBC.py", line 99, in bert_embedding
    use_one_hot_embeddings=False)
  File "/root/Tai/Project/ANER-规范性文件清单/albert/modeling.py", line 207, in __init__
    dropout_prob=config.hidden_dropout_prob)
  File "/root/Tai/Project/ANER-规范性文件清单/albert/modeling.py", line 552, in embedding_postprocessor
    token_type_embeddings = tf.matmul(one_hot_ids, token_type_table)
  File "/root/anaconda3/envs/ner/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "/root/anaconda3/envs/ner/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 2647, in matmul
    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
  File "/root/anaconda3/envs/ner/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5925, in mat_mul
    name=name)
  File "/root/anaconda3/envs/ner/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/root/anaconda3/envs/ner/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/root/anaconda3/envs/ner/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/root/anaconda3/envs/ner/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
```
</details>