tf.test.gpu_device_name() leads to soft lockup and unusable system
Hi @mjugl ,

I think the problem is not with Tensorflow.I tested the same code in colab and it executes fine.[Gist](https://colab.research.google.com/gist/SuryanarayanaY/39ae6bc8ded874147e9ef57171a6111f/60900.ipynb) attached for reference.

I request you to please refer the attached pip instructions guide to check the Hardware requirements and GPU setup process. I have seen from the attached logs GPU was not enabled which may be causing the problem.Please verify the GPU steps and confirm the same. 

Coming to GPU drivers you can check the Nvidia website for compatible GPU driver for your GPU.Also Tensorflow tested configurations for CUDA and cuDNN can be found [here](https://www.tensorflow.org/install/source#gpu).

Thanks! Hi @SuryanarayanaY,

> I have seen from the attached logs GPU was not enabled which may be causing the problem.Please verify the GPU steps and confirm the same.

I reinstalled the graphics drivers and verified they work. `nvidia-smi` reports that the GPU is indeed active. I'm also fairly confident the graphics drivers were correctly installed before. Unfortunately, I can only provide a screenshot of the CLI for now.

![Screenshot_20230619_114937](https://github.com/tensorflow/tensorflow/assets/120371336/63f317c0-1cd1-4ba8-b0af-bccfb8204e09)

> Also Tensorflow tested configurations for CUDA and cuDNN can be found [here](https://www.tensorflow.org/install/source#gpu).

Good point. I found that we're running CUDA 11.4 which is not officially tested with TF 2.12. After the graphics driver re-installation I set up a fresh Tensorflow install as shown in the docs, this time with TF 2.11. I'm getting the same soft lockup (and an unresponsive server) when performing the "Verify install" step.

![Screenshot_20230619_120227](https://github.com/tensorflow/tensorflow/assets/120371336/b60984c6-54e7-4227-b42e-8f704dae2c97)

You can also see that it is being shown in the process list of the `nvidia-smi` screenshot above. Okay I just verified that this issue is not unique to Tensorflow. I ran the bandwidth test from the CUDA samples repository and found it also leads to a CPU soft lockup. So I can confirm that this is likely not an issue with Tensorflow.

On the other hand, I've no idea where to go from here. The issue persists and I've no idea what to look for next. Hi @mjugl ,

I am also quite confident that the issue might be related to your own environment as it is not replicable from our side. I have tried it on GCP VM and also can get desired result as per logs attached below.

```
(bazel) suryanarayanay@surya-ubuntu20:~$ python3 -c "import tensorflow as tf; print(tf.test.gpu_device_name())"
2023-06-28 05:24:05.535235: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-28 05:24:06.474920: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-06-28 05:24:09.452020: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:24:09.499115: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:24:09.500874: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:24:11.554723: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:24:11.556761: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:24:11.558432: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:24:11.560108: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /device:GPU:0 with 13623 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
/device:GPU:0
(bazel) suryanarayanay@surya-ubuntu20:~$ python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
2023-06-28 05:26:39.033522: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-28 05:26:39.988490: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-06-28 05:26:42.978403: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:26:43.025696: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:26:43.027424: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:26:43.029879: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:26:43.031511: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:26:43.033072: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:26:44.111462: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:26:44.113411: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:26:44.115093: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-28 05:26:44.116645: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13623 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
tf.Tensor(1441.14, shape=(), dtype=float32)
(bazel) suryanarayanay@surya-ubuntu20:~$ 
```

The issue is specific to your own environment which I even don't have a clue on this.I believe the issue might be related to OS. I am attaching few references from Internet for similar issues which are related to OS.Please refer [link1](https://askubuntu.com/questions/1264859/watchdog-bug-soft-lockup-cpu6-stuck-for-23s) and [link2](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1989521). 

Please refer the above links and maybe you can get more help from internet community. As the issues is not relevant to TF can we mark it as closed. 

Thanks! Hi, thanks for investigating this further. The issue has been resolved and it didn't have anything to do with Tensorflow in the end.

For anyone who might see this in the future: our IT department forgot to inform us that the licensing process to access the GPU has changed very recently and that we needed to make some modification to our guest VM. The most likely explanation is that the GPU slows down after extended unlicensed use and probably stops working entirely, leading to the lockup we experienced.

We haven't encountered this issue since we reconfigured our guest VM. Again thanks so much for taking the time to investigate even though this really was an issue on our behalf. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60900">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60900">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

2.12.0

### Custom Code

Yes

### OS Platform and Distribution

Linux Ubuntu 20.04

### Mobile device

_No response_

### Python version

3.8.16 (Conda 11.4)

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

11.4

### GPU model and memory

NVIDIA T4

### Current Behaviour?

We found that running `tf.test.gpu_device_name()` leads to a soft lockup and an unresponsive system.

I apologise in advance, I don't know much about Tensorflow. But several of my co-workers do and I'm in charge of maintaining infrastructure. We have a virtual server for compute-intensive tasks where we train models and work with large datasets. Yesterday at about 7 local time I found that I couldn't SSH into the server so I had to wait for IT to forcibly restart the server. At around 13 local time the server was back up and I looked through the kernel logs to find that there was a soft lockup kernel bug. This means that the server was still running, but some process wasn't releasing the CPU for more than 20 seconds, which meant that no other process could fulfill its tasks.

Not even half an hour later and the server locked up again. I got a message from a co-worker who suspected that they were at fault for the server locking up because they started running a very compute-intensive Python script on the server last night. They opened a Python shell and typed in these two lines and watched the server lock up in real time.

```py
import tensorflow as tf 
print('Default GPU Device{}'.format(tf.test.gpu_device_name()))
```

I provided the relevant log output below. It required another hard reset from IT to get the server back into a working state.

After the server restarted again I had another look at the kernel logs and found that the reported errors were the same, which means I have reason to believe that running `tf.test.gpu_device_name()` consistently leads to a soft lockup on our infrastructure.

As mentioned previously, we run our computations on a virtual server. The hypervisor is VMWare. We're running on a Nvidia T4 GPU with Nvidia drivers in version 470.63.01. These drivers were provided to us by IT. Installing another version, for some reason, isn't possible and I prefer not to mess with that. The co-worker installed Tensorflow in a Conda environment using Pip.

Since we're running on a virtual server, we depend on IT in case things go very wrong. This means our means of reproducing this issue are somewhat limited. If the server locks up, we have to wait up to a couple hours for our service desk to escalate this issue enough so that IT restarts the server. We cannot afford downtimes like these since we rely on the server for our computations. I'm happy to provide any logs from previous runs but if possible I would like to not share them on GitHub in their entirety since they might contain confidential info.

### Standalone code to reproduce the issue

```shell
import tensorflow as tf
print('Default GPU Device {}'.format(tf.test.gpu_device_name()))
```


### Relevant log output

```shell
# Output from running the function in a Python shell

$ python3
Python 3.8.16 (default, Mar  2 2023, 03:21:46) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
import tensorflow as tf 
2023-06-15 13:34:49.940226: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-06-15 13:34:50.367115: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-06-15 13:34:50.368678: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-15 13:34:52.102738: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
>>> print('Default GPU Device{}'.format(tf.test.gpu_device_name()))
2023-06-15 13:35:05.586532: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-15 13:35:05.587485: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1956] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

Message from syslogd@s050009088 at Jun 15 13:35:32 ...
 kernel:[  176.957689] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [python3:4167]


# Output from journalctl

Jun 15 13:35:05 s050009088 kernel: nvidia-uvm: Loaded the UVM driver, major device number 238.
Jun 15 13:35:05 s050009088 kernel: ------------[ cut here ]------------
Jun 15 13:35:05 s050009088 kernel: Trying to vfree() nonexistent vm area (00000000f9521180)
Jun 15 13:35:05 s050009088 kernel: WARNING: CPU: 6 PID: 4167 at mm/vmalloc.c:2245 __vunmap+0x1ff/0x210
Jun 15 13:35:05 s050009088 kernel: Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype iptable_filter bpfilter br_netfilter bridge stp llc aufs overlay vmw_vsock_vmci_transport vsock dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua binfmt_misc nvidia_drm(POE) nvidia_modeset(POE) intel_rapl_msr nvidia(POE) vmw_balloon intel_rapl_common input_leds intel_powerclamp joydev rapl serio_raw vmw_vmci mac_hid sch_fq_codel msr ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear vmwgfx crct10dif_pclmul crc32_pclmul ttm ghash_clmulni_intel drm_kms_helper syscopyarea aesni_intel sysfillrect crypto_simd sysimgblt mptspi cryptd fb_sys_fops glue_helper psmouse mptscsih drm mptbase ahci i2c_piix4 vmxnet3 scsi_transport_spi
Jun 15 13:35:05 s050009088 kernel:  libahci pata_acpi
Jun 15 13:35:05 s050009088 kernel: CPU: 6 PID: 4167 Comm: python3 Tainted: P           OE     5.4.0-150-generic #167-Ubuntu
Jun 15 13:35:05 s050009088 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Jun 15 13:35:05 s050009088 kernel: RIP: 0010:__vunmap+0x1ff/0x210
Jun 15 13:35:05 s050009088 kernel: Code: ff e8 d5 fc ff ff eb bf 48 89 fe 48 c7 c7 a0 6d b8 8e e8 36 ba 82 00 0f 0b eb b4 4c 89 ee 48 c7 c7 c8 6d b8 8e e8 23 ba 82 00 <0f> 0b eb a1 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00
Jun 15 13:35:05 s050009088 kernel: RSP: 0018:ffffab66c32b7c00 EFLAGS: 00010286
Jun 15 13:35:05 s050009088 kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000006
Jun 15 13:35:05 s050009088 kernel: RDX: 0000000000000007 RSI: 0000000000000096 RDI: ffff8ce3bfb9c8c0
Jun 15 13:35:05 s050009088 kernel: RBP: ffffab66c32b7c28 R08: 00000000000006d2 R09: 286565726676206f
Jun 15 13:35:05 s050009088 kernel: R10: 286565726676206f R11: 6978656e6f6e2029 R12: 0000000000000000
Jun 15 13:35:05 s050009088 kernel: R13: ffff8ce22c348000 R14: ffff8ce22c349000 R15: 0000000000000027
Jun 15 13:35:05 s050009088 kernel: FS:  00007fbf935e3180(0000) GS:ffff8ce3bfb80000(0000) knlGS:0000000000000000
Jun 15 13:35:05 s050009088 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 15 13:35:05 s050009088 kernel: CR2: 000000000435c3d0 CR3: 0000001e50400001 CR4: 00000000003606e0
Jun 15 13:35:05 s050009088 kernel: Call Trace:
Jun 15 13:35:05 s050009088 kernel:  __vfree+0x22/0x60
Jun 15 13:35:05 s050009088 kernel:  vfree+0x2c/0x40
Jun 15 13:35:05 s050009088 kernel:  os_free_mem+0x1b/0x30 [nvidia]
Jun 15 13:35:05 s050009088 kernel:  os_unlock_user_pages+0x6c/0xa0 [nvidia]
Jun 15 13:35:05 s050009088 kernel:  _nv000647rm+0xdc/0x160 [nvidia]
Jun 15 13:35:05 s050009088 kernel: WARNING: kernel stack frame pointer at 0000000068b18731 in python3:4167 has bad value 00000000f3874266
Jun 15 13:35:05 s050009088 kernel: unwind stack type:0 next_sp:0000000000000000 mask:0x2 graph_idx:0
Jun 15 13:35:05 s050009088 kernel: 00000000c0e190e6: ffffab66c32b7c40 (0xffffab66c32b7c40)
Jun 15 13:35:05 s050009088 kernel: 0000000074c2bd79: ffffffff8da6e972 (__vfree+0x22/0x60)
Jun 15 13:35:05 s050009088 kernel: 00000000ea3bef41: 0000000000000200 (0x200)
Jun 15 13:35:05 s050009088 kernel: 00000000d32e6150: ffffab66c32b7c58 (0xffffab66c32b7c58)
Jun 15 13:35:05 s050009088 kernel: 00000000e4c2b257: ffffffff8da6e9dc (vfree+0x2c/0x40)
Jun 15 13:35:05 s050009088 kernel: 000000008b622c26: ffff8ce22c348000 (0xffff8ce22c348000)
Jun 15 13:35:05 s050009088 kernel: 00000000b443bda0: ffffab66c32b7c68 (0xffffab66c32b7c68)
Jun 15 13:35:05 s050009088 kernel: 0000000004f89fc7: ffffffffc068ccdb (os_free_mem+0x1b/0x30 [nvidia])
Jun 15 13:35:05 s050009088 kernel: 0000000058382150: ffffab66c32b7c98 (0xffffab66c32b7c98)
Jun 15 13:35:05 s050009088 kernel: 000000007fb1b873: ffffffffc068edfc (os_unlock_user_pages+0x6c/0xa0 [nvidia])
Jun 15 13:35:05 s050009088 kernel: 00000000629b0f28: ffff8ce395ff60c8 (0xffff8ce395ff60c8)
Jun 15 13:35:05 s050009088 kernel: 000000003a4ac7ef: 0000010004400000 (0x10004400000)
Jun 15 13:35:05 s050009088 kernel: 000000005e9a76c9: 0000000000000200 (0x200)
Jun 15 13:35:05 s050009088 kernel: 000000005fe4a88b: 00000000aa000000 (0xaa000000)
Jun 15 13:35:05 s050009088 kernel: 0000000068b18731: ffff8ce3b3e92fb0 (0xffff8ce3b3e92fb0)
Jun 15 13:35:05 s050009088 kernel: 00000000bd345bcf: ffffffffc101e8dc (_nv000647rm+0xdc/0x160 [nvidia])
Jun 15 13:35:05 s050009088 kernel: 0000000089f90db7: ffff8ce395ff60c8 (0xffff8ce395ff60c8)
Jun 15 13:35:05 s050009088 kernel: 00000000c99c1977: ffff8ce39726c0c0 (0xffff8ce39726c0c0)
Jun 15 13:35:05 s050009088 kernel: 000000009060a662: 0000000000000008 (0x8)
Jun 15 13:35:05 s050009088 kernel: 0000000043ee1352: ffffffffc101f3a3 (_nv000723rm+0xa43/0xa90 [nvidia])
Jun 15 13:35:05 s050009088 kernel: 0000000064b64c73: 0000000000000000 ...
Jun 15 13:35:05 s050009088 kernel: 0000000030c878cd: ffff8ce3b35bc000 (0xffff8ce3b35bc000)
Jun 15 13:35:05 s050009088 kernel: 00000000e60fac70: ffffffffc101f316 (_nv000723rm+0x9b6/0xa90 [nvidia])
Jun 15 13:35:05 s050009088 kernel: 0000000065747b0c: ffff8ce3b3e90000 (0xffff8ce3b3e90000)
Jun 15 13:35:05 s050009088 kernel: 0000000063da2df1: ffff8ce3a658b800 (0xffff8ce3a658b800)
Jun 15 13:35:05 s050009088 kernel: 0000000008c0ac4e: ffffab66c32b7e48 (0xffffab66c32b7e48)
Jun 15 13:35:05 s050009088 kernel: 0000000057ee515b: ffff8ce3b35b9400 (0xffff8ce3b35b9400)
Jun 15 13:35:05 s050009088 kernel: 000000006c5468b7: 0000000000000027 (0x27)
Jun 15 13:35:05 s050009088 kernel: 0000000065281df3: ffffffffc1025204 (rm_ioctl+0x54/0xb0 [nvidia])
Jun 15 13:35:05 s050009088 kernel: 00000000b419e233: 000000388da5fdb5 (0x388da5fdb5)
Jun 15 13:35:05 s050009088 kernel: 000000006f1b29a3: ffff8ce39726c0c0 (0xffff8ce39726c0c0)
Jun 15 13:35:05 s050009088 kernel: 00000000465f1196: 8000000000000027 (0x8000000000000027)
Jun 15 13:35:05 s050009088 kernel: 0000000024eadd83: 0000000000001047 (0x1047)
Jun 15 13:35:05 s050009088 kernel: 00000000e74dc1ab: 00000000000007e9 (0x7e9)
Jun 15 13:35:05 s050009088 kernel: 00000000f6d262ce: 003d08dcf6746300 (0x3d08dcf6746300)
Jun 15 13:35:05 s050009088 kernel: 00000000400760c5: 003d08dde4df8b00 (0x3d08dde4df8b00)
Jun 15 13:35:05 s050009088 kernel: 00000000c7874ca7: 003d08e3f2980f00 (0x3d08e3f2980f00)
Jun 15 13:35:05 s050009088 kernel: 00000000946d2608: 003d08dd6da9f700 (0x3d08dd6da9f700)
Jun 15 13:35:05 s050009088 kernel: 000000007a9cecb7: 0000000000000000 ...
Jun 15 13:35:05 s050009088 kernel: 000000000e6b2400: 0000000000000200 (0x200)
Jun 15 13:35:05 s050009088 kernel: 0000000022df682d: 0000002000000006 (0x2000000006)
Jun 15 13:35:05 s050009088 kernel: 0000000005972aee: 0000000000001047 (0x1047)
Jun 15 13:35:05 s050009088 kernel: 00000000f3ab8670: 0000000000000000 ...
Jun 15 13:35:05 s050009088 kernel: 0000000043395ee6: 0000000000000001 (0x1)
Jun 15 13:35:05 s050009088 kernel: 0000000090b81e91: 0000000000000000 ...
Jun 15 13:35:05 s050009088 kernel: 00000000ce75a231: ffff8ce39726c0c0 (0xffff8ce39726c0c0)
Jun 15 13:35:05 s050009088 kernel: 00000000336aa9c4: 0000000000000038 (0x38)
Jun 15 13:35:05 s050009088 kernel: 0000000016e174a9: ffff8ce3b35b9400 (0xffff8ce3b35b9400)
Jun 15 13:35:05 s050009088 kernel: 00000000c453e6ee: ffff8ce3a658b800 (0xffff8ce3a658b800)
Jun 15 13:35:05 s050009088 kernel: 00000000ca8fc06a: 0000000000000000 ...
Jun 15 13:35:05 s050009088 kernel: 0000000027388a61: ffffffffc068268f (nvidia_ioctl+0x66f/0x880 [nvidia])
Jun 15 13:35:05 s050009088 kernel: 000000001f100b3c: ffff8ce3b3e90000 (0xffff8ce3b3e90000)
Jun 15 13:35:05 s050009088 kernel: 00000000d70e2948: 00007ffd116cef60 (0x7ffd116cef60)
Jun 15 13:35:05 s050009088 kernel: 000000009e15ae1f: ffff8ce3b35b9498 (0xffff8ce3b35b9498)
Jun 15 13:35:05 s050009088 kernel: 000000002d53d487: 00007ffd00000027 (0x7ffd00000027)
Jun 15 13:35:05 s050009088 kernel: 000000002ea37e99: 0000000000000000 ...
Jun 15 13:35:05 s050009088 kernel: 00000000723c86e7: 0000000000000003 (0x3)
Jun 15 13:35:05 s050009088 kernel: 000000004b070af4: 3c5520dc9b31e900 (0x3c5520dc9b31e900)
Jun 15 13:35:05 s050009088 kernel: 00000000260adce7: ffff8ce3a37b1e01 (0xffff8ce3a37b1e01)
Jun 15 13:35:05 s050009088 kernel: 00000000f3ab64dd: ffff8ce3a37b1e00 (0xffff8ce3a37b1e00)
Jun 15 13:35:05 s050009088 kernel: 000000009250db0d: ffff8ce3a46feaa0 (0xffff8ce3a46feaa0)
Jun 15 13:35:05 s050009088 kernel: 000000001a439626: 00007ffd116cef60 (0x7ffd116cef60)
Jun 15 13:35:05 s050009088 kernel: 00000000d28c869f: ffff8ce3a37b1e00 (0xffff8ce3a37b1e00)
Jun 15 13:35:05 s050009088 kernel: 00000000264cc6e1: ffffab66c32b7e58 (0xffffab66c32b7e58)
Jun 15 13:35:05 s050009088 kernel: 00000000c9cdb9ef: ffffffffc069191b (nvidia_frontend_unlocked_ioctl+0x3b/0x50 [nvidia])
Jun 15 13:35:05 s050009088 kernel: 00000000dc15a7de: ffffab66c32b7ed8 (0xffffab66c32b7ed8)
Jun 15 13:35:05 s050009088 kernel: 000000009b8c3053: ffffffff8dae8e77 (do_vfs_ioctl+0x407/0x670)
Jun 15 13:35:05 s050009088 kernel: 00000000f2d02675: 0000010004400000 (0x10004400000)
Jun 15 13:35:05 s050009088 kernel: 00000000c52e1241: 0000000000000000 ...
Jun 15 13:35:05 s050009088 kernel: 0000000027ba7e88: ffffab66c32b7e78 (0xffffab66c32b7e78)
Jun 15 13:35:05 s050009088 kernel: 00000000aaf3e429: ffffab66c32b7e78 (0xffffab66c32b7e78)
Jun 15 13:35:05 s050009088 kernel: 00000000fd0b1f6d: 3c5520dc9b31e900 (0x3c5520dc9b31e900)
Jun 15 13:35:05 s050009088 kernel: 0000000051d890d1: 0000000000000031 (0x31)
Jun 15 13:35:05 s050009088 kernel: 000000001e5b54d6: 0000000000200000 (0x200000)
Jun 15 13:35:05 s050009088 kernel: 00000000bf7d038a: 0000000000000000 ...
Jun 15 13:35:05 s050009088 kernel: 00000000a021b6f5: 3c5520dc9b31e900 (0x3c5520dc9b31e900)
Jun 15 13:35:05 s050009088 kernel: 00000000382d5bbd: ffff8ce3a37b1e01 (0xffff8ce3a37b1e01)
Jun 15 13:35:05 s050009088 kernel: 000000005691996c: 0000000000000006 (0x6)
Jun 15 13:35:05 s050009088 kernel: 000000002b0a9fae: 00000000c0384627 (0xc0384627)
Jun 15 13:35:05 s050009088 kernel: 00000000ed0791ed: 00007ffd116cef60 (0x7ffd116cef60)
Jun 15 13:35:05 s050009088 kernel: 00000000944e63ee: ffff8ce3a37b1e00 (0xffff8ce3a37b1e00)
Jun 15 13:35:05 s050009088 kernel: 00000000b8133399: ffffab66c32b7f18 (0xffffab66c32b7f18)
Jun 15 13:35:05 s050009088 kernel: 000000006b019d54: ffffffff8dae9147 (ksys_ioctl+0x67/0x90)
Jun 15 13:35:05 s050009088 kernel: 0000000025959ce7: 3c5520dc9b31e900 (0x3c5520dc9b31e900)
Jun 15 13:35:05 s050009088 kernel: 0000000019e92101: 0000000000000000 ...
Jun 15 13:35:05 s050009088 kernel: 000000002801fc84: ffffab66c32b7f58 (0xffffab66c32b7f58)
Jun 15 13:35:05 s050009088 kernel: 0000000060185eb1: 0000000000000000 ...
Jun 15 13:35:05 s050009088 kernel: 000000007c1db1b7: ffffab66c32b7f28 (0xffffab66c32b7f28)
Jun 15 13:35:05 s050009088 kernel: 0000000084b8d84e: ffffffff8dae918a (__x64_sys_ioctl+0x1a/0x20)
Jun 15 13:35:05 s050009088 kernel: 000000009b8e8f5f: ffffab66c32b7f48 (0xffffab66c32b7f48)
Jun 15 13:35:05 s050009088 kernel: 00000000f2da7b14: ffffffff8d804fd7 (do_syscall_64+0x57/0x190)
Jun 15 13:35:05 s050009088 kernel: 00000000942f1f70: 0000000000000000 ...
Jun 15 13:35:05 s050009088 kernel: 00000000f56a3750: ffffffff8e4000a4 (entry_SYSCALL_64_after_hwframe+0x5c/0xc1)
Jun 15 13:35:05 s050009088 kernel: 0000000018f467fe: 00007ffd116ceed0 (0x7ffd116ceed0)
Jun 15 13:35:05 s050009088 kernel: 00000000831b89bf: 00000000648af769 (0x648af769)
Jun 15 13:35:05 s050009088 kernel: 000000000597aba6: 00007ffd116cef88 (0x7ffd116cef88)
Jun 15 13:35:05 s050009088 kernel: 00000000a139f77e: 0000000000000006 (0x6)
Jun 15 13:35:05 s050009088 kernel: 000000006612db4e: 00000000c0384627 (0xc0384627)
Jun 15 13:35:05 s050009088 kernel: 00000000ab55bfe9: 00007ffd116cef60 (0x7ffd116cef60)
Jun 15 13:35:05 s050009088 kernel: 000000000b644430: 0000000000000246 (0x246)
Jun 15 13:35:05 s050009088 kernel: 00000000d2d8779b: 0000000000000000 ...
Jun 15 13:35:05 s050009088 kernel: 00000000725ab6b6: 00007ffd116cef88 (0x7ffd116cef88)
Jun 15 13:35:05 s050009088 kernel: 000000000399d500: 00007ffd116cef60 (0x7ffd116cef60)
Jun 15 13:35:05 s050009088 kernel: 00000000c0a877c1: ffffffffffffffda (0xffffffffffffffda)
Jun 15 13:35:05 s050009088 kernel: 00000000fb645881: 00007fbf936f83ab (0x7fbf936f83ab)
Jun 15 13:35:05 s050009088 kernel: 00000000f6cd2602: 00007ffd116cef60 (0x7ffd116cef60)
Jun 15 13:35:05 s050009088 kernel: 000000006fc36d91: 00000000c0384627 (0xc0384627)
Jun 15 13:35:05 s050009088 kernel: 00000000e5058b78: 0000000000000006 (0x6)
Jun 15 13:35:05 s050009088 kernel: 000000002777ac42: 0000000000000010 (0x10)
Jun 15 13:35:05 s050009088 kernel: 00000000d68cb82a: 00007fbf936f83ab (0x7fbf936f83ab)
Jun 15 13:35:05 s050009088 kernel: 0000000077e7b064: 0000000000000033 (0x33)
Jun 15 13:35:05 s050009088 kernel: 0000000037ef3830: 0000000000000246 (0x246)
Jun 15 13:35:05 s050009088 kernel: 00000000fcba1669: 00007ffd116ceec8 (0x7ffd116ceec8)
Jun 15 13:35:05 s050009088 kernel: 00000000ea0caaa1: 000000000000002b (0x2b)
Jun 15 13:35:05 s050009088 kernel:  ? _nv000723rm+0xa43/0xa90 [nvidia]
Jun 15 13:35:05 s050009088 kernel:  ? _nv000723rm+0x9b6/0xa90 [nvidia]
Jun 15 13:35:05 s050009088 kernel:  ? rm_ioctl+0x54/0xb0 [nvidia]
Jun 15 13:35:05 s050009088 kernel:  ? nvidia_ioctl+0x66f/0x880 [nvidia]
Jun 15 13:35:05 s050009088 kernel:  ? nvidia_frontend_unlocked_ioctl+0x3b/0x50 [nvidia]
Jun 15 13:35:05 s050009088 kernel:  ? do_vfs_ioctl+0x407/0x670
Jun 15 13:35:05 s050009088 kernel:  ? ksys_ioctl+0x67/0x90
Jun 15 13:35:05 s050009088 kernel:  ? __x64_sys_ioctl+0x1a/0x20
Jun 15 13:35:05 s050009088 kernel:  ? do_syscall_64+0x57/0x190
Jun 15 13:35:05 s050009088 kernel:  ? entry_SYSCALL_64_after_hwframe+0x5c/0xc1
Jun 15 13:35:05 s050009088 kernel: ---[ end trace be1a4a9ea080b7b8 ]---

## The following logs are then repeatedly printed, leading to the soft lockup

Jun 15 13:35:05 s050009088 kernel: BUG: Bad page state in process python3  pfn:1e301a6
Jun 15 13:35:05 s050009088 kernel: page:ffffd052f8c06980 refcount:0 mapcount:0 mapping:ffff8ce230f6e0d8 index:0x1
Jun 15 13:35:05 s050009088 kernel: shmem_aops name:"dev/zero"
Jun 15 13:35:05 s050009088 kernel: flags: 0x17ffffc0000000()
Jun 15 13:35:05 s050009088 kernel: raw: 0017ffffc0000000 dead000000000100 dead000000000122 ffff8ce230f6e0d8
Jun 15 13:35:05 s050009088 kernel: raw: 0000000000000001 0000000000000000 00000000ffffffff 0000000000000000
Jun 15 13:35:05 s050009088 kernel: page dumped because: non-NULL mapping
Jun 15 13:35:05 s050009088 kernel: Modules linked in: nvidia_uvm(OE) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype iptable_filter bpfilter br_netfilter bridge stp llc aufs overlay vmw_vsock_vmci_transport vsock dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua binfmt_misc nvidia_drm(POE) nvidia_modeset(POE) intel_rapl_msr nvidia(POE) vmw_balloon intel_rapl_common input_leds intel_powerclamp joydev rapl serio_raw vmw_vmci mac_hid sch_fq_codel msr ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear vmwgfx crct10dif_pclmul crc32_pclmul ttm ghash_clmulni_intel drm_kms_helper syscopyarea aesni_intel sysfillrect crypto_simd sysimgblt mptspi cryptd fb_sys_fops glue_helper psmouse mptscsih drm mptbase ahci i2c_piix4 vmxnet3 scsi_transport_spi
Jun 15 13:35:05 s050009088 kernel:  libahci pata_acpi
Jun 15 13:35:05 s050009088 kernel: CPU: 6 PID: 4167 Comm: python3 Tainted: P        W  OE     5.4.0-150-generic #167-Ubuntu
Jun 15 13:35:05 s050009088 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Jun 15 13:35:05 s050009088 kernel: Call Trace:
Jun 15 13:35:05 s050009088 kernel:  dump_stack+0x6d/0x8b
Jun 15 13:35:05 s050009088 kernel:  bad_page.cold+0x80/0xb1
Jun 15 13:35:05 s050009088 kernel:  free_pages_check_bad+0x5f/0x70
Jun 15 13:35:05 s050009088 kernel:  free_pcppages_bulk+0x186/0x6b0
Jun 15 13:35:05 s050009088 kernel:  free_unref_page_commit+0xb6/0xd0
Jun 15 13:35:05 s050009088 kernel:  free_unref_page_list+0x107/0x190
Jun 15 13:35:05 s050009088 kernel:  release_pages+0x38d/0x400
Jun 15 13:35:05 s050009088 kernel:  free_pages_and_swap_cache+0xb9/0xd0
Jun 15 13:35:05 s050009088 kernel:  tlb_flush_mmu+0x3a/0x140
Jun 15 13:35:05 s050009088 kernel:  zap_pte_range.isra.0+0x563/0x860
Jun 15 13:35:05 s050009088 kernel:  ? __warn+0x9d/0xe0
Jun 15 13:35:05 s050009088 kernel:  unmap_page_range+0x2e6/0x560
Jun 15 13:35:05 s050009088 kernel:  unmap_single_vma+0x7f/0xf0
Jun 15 13:35:05 s050009088 kernel:  unmap_vmas+0x79/0xf0
Jun 15 13:35:05 s050009088 kernel:  unmap_region+0xbc/0x160
Jun 15 13:35:05 s050009088 kernel:  __do_munmap+0x2aa/0x500
Jun 15 13:35:05 s050009088 kernel:  mmap_region+0x248/0x650
Jun 15 13:35:05 s050009088 kernel:  do_mmap+0x3b4/0x5c0
Jun 15 13:35:05 s050009088 kernel:  vm_mmap_pgoff+0xcb/0x120
Jun 15 13:35:05 s050009088 kernel:  ksys_mmap_pgoff+0x125/0x2b0
Jun 15 13:35:05 s050009088 kernel:  ? fput+0x13/0x20
Jun 15 13:35:05 s050009088 kernel:  ? ksys_ioctl+0x77/0x90
Jun 15 13:35:05 s050009088 kernel:  __x64_sys_mmap+0x33/0x40
Jun 15 13:35:05 s050009088 kernel:  do_syscall_64+0x57/0x190
Jun 15 13:35:05 s050009088 kernel:  entry_SYSCALL_64_after_hwframe+0x5c/0xc1
Jun 15 13:35:05 s050009088 kernel: RIP: 0033:0x7fbf936fc8e6
Jun 15 13:35:05 s050009088 kernel: Code: 00 00 00 00 f3 0f 1e fa 41 f7 c1 ff 0f 00 00 75 2b 55 48 89 fd 53 89 cb 48 85 ff 74 37 41 89 da 48 89 ef b8 09 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 62 5b 5d c3 0f 1f 80 00 00 00 00 48 8b 05 71
Jun 15 13:35:05 s050009088 kernel: RSP: 002b:00007ffd116cf078 EFLAGS: 00000206 ORIG_RAX: 0000000000000009
Jun 15 13:35:05 s050009088 kernel: RAX: ffffffffffffffda RBX: 0000000000000032 RCX: 00007fbf936fc8e6
Jun 15 13:35:05 s050009088 kernel: RDX: 0000000000000000 RSI: 0000000000200000 RDI: 0000010004400000
Jun 15 13:35:05 s050009088 kernel: RBP: 0000010004400000 R08: 00000000ffffffff R09: 0000000000000000
Jun 15 13:35:05 s050009088 kernel: R10: 0000000000000032 R11: 0000000000000206 R12: 00000000044be5e0
Jun 15 13:35:05 s050009088 kernel: R13: 0000000000000001 R14: 00000000047d97c0 R15: 000000000430b450

## Eventually the watchdog reports the soft lockup
## Checking the PID against the one from the process that ran `tf.test.gpu_device_name()` shows that it must've caused this lockup

Jun 15 13:46:45 s050009088 kernel: watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [python3:4167]
```
</details>