could not run GPU on jupyter 
ther could be few reasons but the main is to Ensure that GPU drivers are installed und make sur you have the appropriate GPU drivers on your system. Hi @gg4u ,

Could you please confirm the OS you are using. Could you please confirm whether GPU drivers and nvidia toolkit installed and path variables set properly ? Refer to the documentation source [here](https://www.tensorflow.org/install/pip#step-by-step_instructions) and confirm this was followed.

Please share the output of `nvidia-smi` command and `python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"` from Jupyter. 

The error seems to be related to CUDA path setting. Can you try fresh installation of conda environment on Jupyter notebook and let us know the outcome. OS => I don't know, I am using a university cluster, it must be linux but i don't know which.

Note that I was eventually able to run on jupyter as well, but only if :
I reset :
```
CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/:$CUDNN_PATH/lib
```

and
```
# Install NVCC
conda install -c nvidia cuda-nvcc=11.3.58
# Configure the XLA cuda directory
mkdir -p $CONDA_PREFIX/etc/conda/activate.d
printf 'export XLA_FLAGS=--xla_gpu_cuda_data_dir=$CONDA_PREFIX/lib/\n' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
source $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
# Copy libdevice file to the required path
mkdir -p $CONDA_PREFIX/lib/nvvm/libdevice
cp $CONDA_PREFIX/lib/libdevice.10.bc $CONDA_PREFIX/lib/nvvm/libdevice/
```

and when I launch jupyter lab, it must be launched from the same conda environment I re-set  up those script.
Kernel selection won't work anyway (in other words, I must launch jupyter lab from the same conda environment I intend to use the kernel, otherwise environment variables makes confusion and on jupyter Cuda won't be found).


nvidia-smi:

```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:03:00.0 Off |                  N/A |
| 28%   30C    P8    15W / 250W |  10451MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  On   | 00000000:21:00.0 Off |                  N/A |
| 28%   27C    P8    16W / 250W |   9961MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  On   | 00000000:41:00.0 Off |                  N/A |
| 28%   28C    P8     4W / 250W |   9961MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  On   | 00000000:61:00.0 Off |                  N/A |
| 28%   28C    P8    20W / 250W |   9961MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A   3115300      C   ...3/envs/conda01/bin/python    10448MiB |
|    1   N/A  N/A   3115300      C   ...3/envs/conda01/bin/python     9958MiB |
|    2   N/A  N/A   3115300      C   ...3/envs/conda01/bin/python     9958MiB |
|    3   N/A  N/A   3115300      C   ...3/envs/conda01/bin/python     9958MiB |
+-----------------------------------------------------------------------------+
```

From jupyter:
```

python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

2023-06-07 15:11:35.922385: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-07 15:11:36.765039: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:3', device_type='GPU')]



```

> Can you try fresh installation of conda environment on Jupyter notebook and let us know the outcome.

I did it all yesterday, I spent a whole day, and did so many times.

I won't touch again the installation.

But to reproduce, I did the following:

- create a conda env (as per instruction) and CUDA drivers
- installed ikernel (to select the kernel in jupyter)
- on terminal, TF read the GPU => OK!
- on jupyter lab, I could select a different kernel, BUT inspecting on `!conda -info` and `!which  python` it was keep referring to the base from where jupyter was launched, and that I pervously set the environment variables automatically (see '# script' below from https://www.tensorflow.org/install/pip). 


It was messy to debug, because there were conflicts on `libdevice` in jupter, but not in terminal; probably because the environment variable instructing on the cuda libraries could not be found. Maybe some issues with `ikernel` - I don't have clue only guessing.

I eventually got it to work without automating the variables :

```
# script

mkdir -p $CONDA_PREFIX/etc/conda/activate.d
echo 'CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/:$CUDNN_PATH/lib' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh

```
but setting them manually, and then only launch the jupter lab from the conda environemnt corresponding to the kernel I will work with.


 The attached commands from Documentation should fix system paths to be automatically configured when you activate the conda environment.

I hope once you have created the conda environment and followed all the steps mentioned in documentation including the commands for Automatic enabling of system paths every time conda environment activated should fix the issue. This should be one time exercise only and then it should work every time when activated the respective conda environment.

 Unfortunately not, I see a mismatch between the conda or python (`which python`) listed in the jupyter notebook, and the selected kernel.

Awkwardly, I was able to get TF see the GPUs on Jupyter, only if I created a virtual environment in python (`python -m venv ./env`), and then tried to install cuda drivers from conda , but all other libraries via pip.

However, there are always challenges, because, for example, some libraries, like ones using `cudf`, will fail (I ve just discovered that with aweful regret). 

IT would be highly beneficial, in my opinion, if you could add a guide to install jupyter lab with multiple kernels, so to minismize error for users approaching TF on rermote cloud for first time. Assume people using remote servers. At my university, the help desk even asked me if it was necessary to use jupyter or conda virtual environment (which I actually would be .. knida necessary) to minimise error, and just keep one base - the default. But I am afraid that if I messy up something, I m kinda done.
 What challenges are you facing with ```cuDF```? For resolving dependency issues with TF installed in a conda environment you can try creating a new environment with an earlier version of Python (```conda create -n EnvNameHere python=3.8```) or using tools like Libmamba (see https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community/). Also, try running ```lsb_release -a``` or ```uname -a``` from root outside of any virtual environment. Knowing more about your OS is crucial to determining whether this is a CUDA path issue. @gg4u ,

Could you please provide the OS details also.Thanks! Hi @ggfu,

You might need to use  `conda install nb_conda_kernels` and may be also `conda install ipykernel` additionally for Jupyter.
Could you please follow the instructions mentioned here at [SO1](https://stackoverflow.com/questions/76101948/tensorflow-gpu-recognized-in-the-terminal-but-not-in-the-jupyter-notebook) and [SO2](https://stackoverflow.com/questions/57332001/import-tensorflow-working-in-terminal-but-not-in-jupyter-notebook) and let us know it works.

 Hi @gg4u this is an issue with using Jupyter Notebook itself. You can take a look at this [issue](https://stackoverflow.com/questions/76101948/tensorflow-gpu-recognized-in-the-terminal-but-not-in-the-jupyter-notebook) on why jupyter doesn't call GPU and follow the work around. As for the documentation, tensorflow doesn't officially support Jupyter notebooks and I don't think this will be documented anywhere officially. Thanks!! Hi, thanks for the support. 

I can't summarize exactly what causes are responsible of the issues, I think the problem is conflicting environment variables.

I had this constrains:
- using a university cluster, the university discouraging in using conda env, just installed on the dedicated env
- tensorflow encorages in using conda
- external libraries complained if on conda (installs with pip conflicted).

I temporarilly have resolved in creating a pip envionrment, and launching jupyter lab from there.
Jupyter sees the GPUs, while if it was launched from conda, then TF was not seeing the GPU.
Pytorch saw the GPUs on both.

I followed :
https://stackoverflow.com/questions/76101948/tensorflow-gpu-recognized-in-the-terminal-but-not-in-the-jupyter-notebook

but for some reasons, `which python` on jupyter opened in a chosen  enviroment, was anyway pointing to the environment from where jupyter lab server was launched. Hi @gg4u, Unfortunately there is not much that can be done about this from our end. Please follow the process as mentioned above and this issue should be resolved.
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

2.12.0

### Custom Code

Yes

### OS Platform and Distribution

_No response_

### Mobile device

_No response_

### Python version

3.10.11

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

I could not get Tensorflow to run on GPUs.

TF sees the GPUs on terminal, but not on jupyter lab.

**Edited**
Found a solution to see it on jupyterlab, but must manually repeat . Still, erratic misconfiguration seems to happen.


### Standalone code to reproduce the issue

```shell
I could not get Tensorflow to run on GPUs.

TF sees the GPUs on terminal, but not on jupyter lab.

** edited **


Eventually, after hours, I found a temporary solution in setting the paths each time, before I launch `jupyter lab` :



CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/:$CUDNN_PATH/lib

```

NB : If I was to set 
```
mkdir -p $CONDA_PREFIX/etc/conda/activate.d
echo 'CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/:$CUDNN_PATH/lib' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
```

as in `https://www.tensorflow.org/install/pip`

Somehow it messy with jupyter, bcause the selected kernel from jupyter would not correspond to the kernel set from the script.

Now I can see the GPUs on jupyter, but still - it crashes! And before, on CPU, it was not.

When I run this script, using this library :

https://pypi.org/project/umap-learn/

installed as in the description:

```
embedder = ParametricUMAP(
    ## all the params ...
)

# now launch on GPU
with tf.device('/GPU:0'):
    embedding =  embedder.fit_transform(np.array([t.ravel() for t in train_data]))
```

the code fails with the log output below.

If I close the jupyter lab connection, go back on the conda environment, 
set again:

```
CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/:$CUDNN_PATH/lib
```
I
and 

```
# Install NVCC
conda install -c nvidia cuda-nvcc=11.3.58
# Configure the XLA cuda directory
mkdir -p $CONDA_PREFIX/etc/conda/activate.d
printf 'export XLA_FLAGS=--xla_gpu_cuda_data_dir=$CONDA_PREFIX/lib/\n' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
source $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
# Copy libdevice file to the required path
mkdir -p $CONDA_PREFIX/lib/nvvm/libdevice
cp $CONDA_PREFIX/lib/libdevice.10.bc $CONDA_PREFIX/lib/nvvm/libdevice/

```

And relaunch jupterlab, this time I get the GPU seen also in jupyter lab.

Running 

```
with tf.device('/GPU:0'):
    spec_embedding = spec_embedder.fit_transform(np.array([t.ravel() for t in train_data]))

```

It yields

```
2023-06-06 21:55:04.944257: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'gradients/split_grad/concat/split/split_dim' with dtype int32
	 [[{{node gradients/split_grad/concat/split/split_dim}}]]
```

And cannot understand why it raise warning or error with `CPU` device, if I run on `GPU`. 

Not even sure it falled back to CPU, actually.


---

Please advice how to sync jupter lab and conda.
I did follow up with `ikernel` but it seems, after a lot of checking, that environment variables are not correctly read. Not sure if `kernel.json` fails to be updated properly, with thepath of the cuda libraries.

Please consider add a guide on:
https://www.tensorflow.org/install/pip

for running TF on jupyter.

My situation is that I need to run from a remote cluster, and I think it is a frequent situation.
hope this feedback is useful.
```


### Relevant log output

```shell
2023-06-06 21:40:08.385763: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'gradients/split_grad/concat/split/split_dim' with dtype int32
	 [[{{node gradients/split_grad/concat/split/split_dim}}]]
2023-06-06 21:40:10.938429: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:424] Loaded cuDNN version 8600
2023-06-06 21:40:11.641291: I tensorflow/compiler/xla/service/service.cc:169] XLA service 0x7f89f489b8d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-06-06 21:40:11.641337: I tensorflow/compiler/xla/service/service.cc:177]   StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2023-06-06 21:40:11.641346: I tensorflow/compiler/xla/service/service.cc:177]   StreamExecutor device (1): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2023-06-06 21:40:11.641353: I tensorflow/compiler/xla/service/service.cc:177]   StreamExecutor device (2): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2023-06-06 21:40:11.641359: I tensorflow/compiler/xla/service/service.cc:177]   StreamExecutor device (3): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2023-06-06 21:40:11.646324: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2023-06-06 21:40:11.673008: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:530] Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice.
Searched for CUDA in the following directories:
  ./cuda_sdk_lib
  /usr/local/cuda-11.8
  /usr/local/cuda
  .
You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule's DebugOptions.  For most apps, setting the environment variable XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda will work.
2023-06-06 21:40:11.673259: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:11.673658: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:11.673685: I tensorflow/core/common_runtime/executor.cc:1197] [/job:localhost/replica:0/task:0/device:GPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INTERNAL: libdevice not found at ./libdevice.10.bc
	 [[{{node StatefulPartitionedCall_16}}]]
2023-06-06 21:40:11.700045: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:11.700414: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:11.729242: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:11.729590: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:11.755959: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:11.756308: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:11.783041: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:11.783397: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:11.809786: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:11.810134: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:11.836777: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:11.837129: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:11.864411: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:11.864767: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:11.892388: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:11.892738: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:11.919296: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:11.919647: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:11.946506: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:11.946866: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:11.974287: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:11.974699: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:12.245782: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:12.246188: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:12.272303: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:12.272657: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:12.322559: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:12.322899: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:12.350255: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:12.350736: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:12.379576: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:12.379966: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-06-06 21:40:12.406780: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:274] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-06-06 21:40:12.407233: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: libdevice not found at ./libdevice.10.bc

---------------------------------------------------------------------------
InternalError                             Traceback (most recent call last)
Cell In[33], line 31
      7 spec_embedder = ParametricUMAP(
      8     metric = 'euclidean',
      9     min_dist = 0.1, 
   (...)
     27     n_training_epochs=1
     28 )
     30 with tf.device('/GPU:0'):
---> 31     spec_embedding = spec_embedder.fit_transform(np.array([t.ravel() for t in train_data]))

File ~/miniconda3/envs/conda01/lib/python3.10/site-packages/umap/parametric_umap.py:217, in ParametricUMAP.fit_transform(self, X, y, precomputed_distances)
    215     return super().fit_transform(precomputed_distances, y)
    216 else:
--> 217     return super().fit_transform(X, y)

File ~/miniconda3/envs/conda01/lib/python3.10/site-packages/umap/umap_.py:2772, in UMAP.fit_transform(self, X, y)
   2742 def fit_transform(self, X, y=None):
   2743     """Fit X into an embedded space and return that transformed
   2744     output.
   2745 
   (...)
   2770         Local radii of data points in the embedding (log-transformed).
   2771     """
-> 2772     self.fit(X, y)
   2773     if self.transform_mode == "embedding":
   2774         if self.output_dens:

File ~/miniconda3/envs/conda01/lib/python3.10/site-packages/umap/parametric_umap.py:202, in ParametricUMAP.fit(self, X, y, precomputed_distances)
    200     return super().fit(precomputed_distances, y)
    201 else:
--> 202     return super().fit(X, y)

File ~/miniconda3/envs/conda01/lib/python3.10/site-packages/umap/umap_.py:2684, in UMAP.fit(self, X, y)
   2681     print(ts(), "Construct embedding")
   2683 if self.transform_mode == "embedding":
-> 2684     self.embedding_, aux_data = self._fit_embed_data(
   2685         self._raw_data[index],
   2686         self.n_epochs,
   2687         init,
   2688         random_state,  # JH why raw data?
   2689     )
   2690     # Assign any points that are fully disconnected from our manifold(s) to have embedding
   2691     # coordinates of np.nan.  These will be filtered by our plotting functions automatically.
   2692     # They also prevent users from being deceived a distance query to one of these points.
   2693     # Might be worth moving this into simplicial_set_embedding or _fit_embed_data
   2694     disconnected_vertices = np.array(self.graph_.sum(axis=1)).flatten() == 0

File ~/miniconda3/envs/conda01/lib/python3.10/site-packages/umap/parametric_umap.py:462, in ParametricUMAP._fit_embed_data(self, X, n_epochs, init, random_state)
    459     validation_data = None
    461 # create embedding
--> 462 history = self.parametric_model.fit(
    463     edge_dataset,
    464     epochs=self.loss_report_frequency * self.n_training_epochs,
    465     steps_per_epoch=steps_per_epoch,
    466     max_queue_size=100,
    467     validation_data=validation_data,
    468     **self.keras_fit_kwargs
    469 )
    470 # save loss history dictionary
    471 self._history = history.history

File ~/.local/lib/python3.10/site-packages/keras/utils/traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     67     filtered_tb = _process_traceback_frames(e.__traceback__)
     68     # To get the full stack trace, call:
     69     # `tf.debugging.disable_traceback_filtering()`
---> 70     raise e.with_traceback(filtered_tb) from None
     71 finally:
     72     del filtered_tb

File ~/miniconda3/envs/conda01/lib/python3.10/site-packages/tensorflow/python/eager/execute.py:52, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     50 try:
     51   ctx.ensure_initialized()
---> 52   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     53                                       inputs, attrs, num_outputs)
     54 except core._NotOkStatusException as e:
     55   if name is not None:

InternalError: Graph execution error:

Detected at node 'StatefulPartitionedCall_16' defined at (most recent call last):
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/ipykernel_launcher.py", line 17, in <module>
      app.launch_new_instance()
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/traitlets/config/application.py", line 992, in launch_instance
      app.start()
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 711, in start
      self.io_loop.start()
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 215, in start
      self.asyncio_loop.run_forever()
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
      self._run_once()
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
      handle._run()
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/asyncio/events.py", line 80, in _run
      self._context.run(self._callback, *self._args)
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 510, in dispatch_queue
      await self.process_one()
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 499, in process_one
      await dispatch(*args)
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 406, in dispatch_shell
      await result
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 729, in execute_request
      reply_content = await reply_content
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 411, in do_execute
      res = shell.run_cell(
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 531, in run_cell
      return super().run_cell(*args, **kwargs)
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
      result = self._run_cell(
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
      result = runner(coro)
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
      coro.send(None)
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
      if await self.run_code(code, result, async_=asy):
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "/tmp/ipykernel_3110734/3903349550.py", line 31, in <module>
      spec_embedding = spec_embedder.fit_transform(np.array([t.ravel() for t in train_data]))
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/umap/parametric_umap.py", line 217, in fit_transform
      return super().fit_transform(X, y)
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/umap/umap_.py", line 2772, in fit_transform
      self.fit(X, y)
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/umap/parametric_umap.py", line 202, in fit
      return super().fit(X, y)
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/umap/umap_.py", line 2684, in fit
      self.embedding_, aux_data = self._fit_embed_data(
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/umap/parametric_umap.py", line 462, in _fit_embed_data
      history = self.parametric_model.fit(
    File "/home/h21/luas6629/.local/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/home/h21/luas6629/.local/lib/python3.10/site-packages/keras/engine/training.py", line 1685, in fit
      tmp_logs = self.train_function(iterator)
    File "/home/h21/luas6629/.local/lib/python3.10/site-packages/keras/engine/training.py", line 1284, in train_function
      return step_function(self, iterator)
    File "/home/h21/luas6629/.local/lib/python3.10/site-packages/keras/engine/training.py", line 1268, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/home/h21/luas6629/.local/lib/python3.10/site-packages/keras/engine/training.py", line 1249, in run_step
      outputs = model.train_step(data)
    File "/home/h21/luas6629/miniconda3/envs/conda01/lib/python3.10/site-packages/umap/parametric_umap.py", line 1150, in train_step
      self.optimizer.apply_gradients(zip(gradients, trainable_vars))
    File "/home/h21/luas6629/.local/lib/python3.10/site-packages/keras/optimizers/optimizer.py", line 1174, in apply_gradients
      return super().apply_gradients(grads_and_vars, name=name)
    File "/home/h21/luas6629/.local/lib/python3.10/site-packages/keras/optimizers/optimizer.py", line 650, in apply_gradients
      iteration = self._internal_apply_gradients(grads_and_vars)
    File "/home/h21/luas6629/.local/lib/python3.10/site-packages/keras/optimizers/optimizer.py", line 1200, in _internal_apply_gradients
      return tf.__internal__.distribute.interim.maybe_merge_call(
    File "/home/h21/luas6629/.local/lib/python3.10/site-packages/keras/optimizers/optimizer.py", line 1250, in _distributed_apply_gradients_fn
      distribution.extended.update(
    File "/home/h21/luas6629/.local/lib/python3.10/site-packages/keras/optimizers/optimizer.py", line 1245, in apply_grad_to_update_var
      return self._update_step_xla(grad, var, id(self._var_key(var)))
Node: 'StatefulPartitionedCall_16'
libdevice not found at ./libdevice.10.bc
	 [[{{node StatefulPartitionedCall_16}}]] [Op:__inference_train_function_4496]
```
</details>