title: Another failed launch of Blas GEMM in TensorFlow
body: <p>I have a problem with running the RNN Guide from <a href="https://www.tensorflow.org/guide/keras/rnn" rel="nofollow noreferrer">TensorFlow</a>. I am on Ubuntu 18.04.3 and I've installed TensorFlow with GPU support through Anaconda3. When I run a code as simple as this:</p>
  
  <pre><code>from __future__ import absolute_import, division, print_function, unicode_literals
  import tensorflow as tf
  import numpy as np
  import os
  import time
  
  # DATASET
  path_to_file = './shakespeare.txt'
  text = open(path_to_file, 'rb').read().decode(encoding='utf-8')
  vocab = sorted(set(text))
  char2idx = {u:i for i, u in enumerate(vocab)}
  idx2char = np.array(vocab)
  text_as_int = np.array([char2idx[c] for c in text])
  seq_length = 100
  char_dataset = tf.data.Dataset.from_tensor_slices(text_as_int)
  sequences = char_dataset.batch(seq_length+1, drop_remainder=True)
  def split_input_target(chunk):
      input_text = chunk[:-1]
      target_text = chunk[1:]
      return input_text, target_text
  dataset = sequences.map(split_input_target)
  BATCH_SIZE = 50 #64
  # Buffer size to shuffle the dataset
  # (TF data is designed to work with possibly infinite sequences,
  # so it doesn't attempt to shuffle the entire sequence in memory. Instead,
  # it maintains a buffer in which it shuffles elements).
  BUFFER_SIZE = 10000
  dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)
  
  # MODEL
  vocab_size = len(vocab) #65
  embedding_dim = 256
  rnn_units = 1024
  def build_model(vocab_size, embedding_dim, rnn_units, batch_size):
      model = tf.keras.Sequential([
      tf.keras.layers.Embedding(vocab_size, embedding_dim,
                              batch_input_shape=[batch_size, None]),
      tf.keras.layers.GRU(rnn_units,
                          return_sequences=True,
                          stateful=True,
                          recurrent_initializer='glorot_uniform'),
      tf.keras.layers.Dense(vocab_size)
      ])
      return model
  model = build_model(
      vocab_size = vocab_size,
      embedding_dim=embedding_dim,
      rnn_units=rnn_units,
      batch_size=BATCH_SIZE)
  
  for input_example_batch, target_example_batch in dataset.take(1):
      a = model(input_example_batch)
      print(a)
  </code></pre>
  
  <p>I get this result:</p>
  
  <pre><code>2019-11-22 12:26:38.175152: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
  2019-11-22 12:26:38.183936: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2019-11-22 12:26:38.184486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
  name: GeForce 930M major: 5 minor: 0 memoryClockRate(GHz): 0.941
  pciBusID: 0000:01:00.0
  2019-11-22 12:26:38.201683: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
  2019-11-22 12:26:38.217994: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
  2019-11-22 12:26:38.226954: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
  2019-11-22 12:26:38.249912: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
  2019-11-22 12:26:38.266853: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
  2019-11-22 12:26:38.283398: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
  2019-11-22 12:26:38.305535: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
  2019-11-22 12:26:38.305704: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2019-11-22 12:26:38.306261: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2019-11-22 12:26:38.306761: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
  2019-11-22 12:26:38.307056: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
  2019-11-22 12:26:38.328890: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2594080000 Hz
  2019-11-22 12:26:38.329724: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55de21a15ca0 executing computations on platform Host. Devices:
  2019-11-22 12:26:38.329751: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
  2019-11-22 12:26:38.365863: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2019-11-22 12:26:38.366560: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55de21a17b00 executing computations on platform CUDA. Devices:
  2019-11-22 12:26:38.366589: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce 930M, Compute Capability 5.0
  2019-11-22 12:26:38.366725: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2019-11-22 12:26:38.367183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
  name: GeForce 930M major: 5 minor: 0 memoryClockRate(GHz): 0.941
  pciBusID: 0000:01:00.0
  2019-11-22 12:26:38.367210: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
  2019-11-22 12:26:38.367221: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
  2019-11-22 12:26:38.367230: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
  2019-11-22 12:26:38.367239: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
  2019-11-22 12:26:38.367248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
  2019-11-22 12:26:38.367257: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
  2019-11-22 12:26:38.367266: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
  2019-11-22 12:26:38.367314: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2019-11-22 12:26:38.367786: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2019-11-22 12:26:38.368220: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
  2019-11-22 12:26:38.368245: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
  2019-11-22 12:26:38.368989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
  2019-11-22 12:26:38.369001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
  2019-11-22 12:26:38.369006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
  2019-11-22 12:26:38.369111: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2019-11-22 12:26:38.369593: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2019-11-22 12:26:38.370053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1422 MB memory) -&gt; physical GPU (device: 0, name: GeForce 930M, pci bus id: 0000:01:00.0, compute capability: 5.0)
  2019-11-22 12:26:40.754438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
  2019-11-22 12:26:41.456357: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
  2019-11-22 12:26:41.816442: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
  2019-11-22 12:26:41.820286: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
  2019-11-22 12:26:41.820332: W tensorflow/stream_executor/stream.cc:1919] attempting to perform BLAS operation using StreamExecutor without BLAS support
  Traceback (most recent call last):
    File "high_level_GRU_1.py", line 56, in &lt;module&gt;
      a = model(input_example_batch)
    File "/home/okami/anaconda3/envs/tensorflow/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 891, in __call__
      outputs = self.call(cast_inputs, *args, **kwargs)
    File "/home/okami/anaconda3/envs/tensorflow/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/sequential.py", line 256, in call
      return super(Sequential, self).call(inputs, training=training, mask=mask)
    File "/home/okami/anaconda3/envs/tensorflow/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 708, in call
      convert_kwargs_to_constants=base_layer_utils.call_context().saving)
    File "/home/okami/anaconda3/envs/tensorflow/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 860, in _run_internal_graph
      output_tensors = layer(computed_tensors, **kwargs)
    File "/home/okami/anaconda3/envs/tensorflow/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 891, in __call__
      outputs = self.call(cast_inputs, *args, **kwargs)
    File "/home/okami/anaconda3/envs/tensorflow/lib/python3.7/site-packages/tensorflow_core/python/keras/layers/core.py", line 1045, in call
      outputs = standard_ops.tensordot(inputs, self.kernel, [[rank - 1], [0]])
    File "/home/okami/anaconda3/envs/tensorflow/lib/python3.7/site-packages/tensorflow_core/python/ops/math_ops.py", line 4077, in tensordot
      ab_matmul = matmul(a_reshape, b_reshape)
    File "/home/okami/anaconda3/envs/tensorflow/lib/python3.7/site-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
      return target(*args, **kwargs)
    File "/home/okami/anaconda3/envs/tensorflow/lib/python3.7/site-packages/tensorflow_core/python/ops/math_ops.py", line 2765, in matmul
      a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
    File "/home/okami/anaconda3/envs/tensorflow/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_math_ops.py", line 6126, in mat_mul
      _six.raise_from(_core._status_to_exception(e.code, message), None)
    File "&lt;string&gt;", line 3, in raise_from
  tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(5000, 1024), b.shape=(1024, 65), m=5000, n=65, k=1024 [Op:MatMul] name: sequential/dense/Tensordot/MatMul/
  </code></pre>
  
  <p>My <strong>nvidia-smi</strong> output:</p>
  
  <pre><code>Fri Nov 22 12:34:23 2019       
  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |===============================+======================+======================|
  |   0  GeForce 930M        On   | 00000000:01:00.0 Off |                  N/A |
  | N/A   40C    P5    N/A /  N/A |    350MiB /  2004MiB |      9%      Default |
  +-------------------------------+----------------------+----------------------+
  
  +-----------------------------------------------------------------------------+
  | Processes:                                                       GPU Memory |
  |  GPU       PID   Type   Process name                             Usage      |
  |=============================================================================|
  |    0      1037      G   /usr/lib/xorg/Xorg                            27MiB |
  |    0      1239      G   /usr/bin/gnome-shell                          48MiB |
  |    0      2706      G   /usr/lib/xorg/Xorg                           115MiB |
  |    0      2923      G   /usr/bin/gnome-shell                          74MiB |
  |    0     17495      G   ...quest-channel-token=6887620846227821225    78MiB |
  |    0     19795      G   gnome-control-center                           1MiB |
  +-----------------------------------------------------------------------------+
  </code></pre>
  
  <p>I tried to google my problem, I've found this:</p>
  
  <blockquote>
    <p><a href="https://stackoverflow.com/questions/43990046/tensorflow-blas-gemm-launch-failed">TensorFlow: Blas GEMM launch failed</a></p>
  </blockquote>
  
  <p>But I dont' have tensorflow_backend.py anywhere in my <strong>~/anaconda3/...</strong> directory. Can I fix this without reinstall TensorFlow? Thanks</p>