NCCLAllReduce fails with CUDA 12.0 and CuDNN 8.8.0 when using MirroredStrategy
Hi @rajveerb, cuda 12.0 is not supported on Ubuntu 18.04. As you are using Ubuntu 18.04 and you need to use a newer version of CUDA. You will need to upgrade your operating system to a newer version that is supported. Alternatively, you can use an older version of CUDA that is compatible with Ubuntu 18.04.  Thank you! @synandi 
The support for CUDA 12.0 in Ubuntu 18.04 is available. Check [this](https://developer.nvidia.com/cuda-12-0-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu). @rajveerb, Apologies for the confusion. We are trying to dig deep into the issue. We will update here soon. Thank you!  Hi, For Tensorflow 2.12 release, we have tested the build configuration with CUDA 11.8 version and CuDNN 8.6 version, same details will be published in the table here https://www.tensorflow.org/install/source#gpu.
Could you please test against the above configurations and let us know if you are still facing an error. Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59890">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59890">No</a>
 > @synandi The support for CUDA 12.0 in Ubuntu 18.04 is available. Check [this](https://developer.nvidia.com/cuda-12-0-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu).

hi, any solution now?
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

v2.12.0-rc0

### Custom Code

No

### OS Platform and Distribution

Linux Ubuntu 18.04

### Mobile device

_No response_

### Python version

3.10.9

### Bazel version

5.3.0

### GCC/Compiler version

9.4.0

### CUDA/cuDNN version

12.0/8.8.0

### GPU model and memory

NVIDIA Tesla V100-SXM2 with 16GB

### Current Behaviour?

```
NCCLAllReduce when used in MirrorStrategy causes a crash.
ReductionToOneDevice in MirrorStrategy works fine.
I believe the drivers, CUDA, and CuDNN installations are not an issue because
NCCLAllReduce works fine when using PyTorch.

I have attached relevant code and log from it.

I have built tensorflow from source and there were no errors generated in build process either.

Using TF nightly does not recognize GPUs so I was not able to reproduce the same bug.
```


### Standalone code to reproduce the issue

[Link](https://github.com/tensorflow/docs/blob/9b5b8b055e29d006b94b5dfc906d774ee4094bae/site/en/tutorials/distribute/custom_training.ipynb) from tensorflow's tutorial which can be used for bug replication.


### Relevant log output

```python
2023-03-03 13:12:14.345311: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 14575 MB memory:  -> device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:04:00.0, compute capability: 7.0
2023-03-03 13:12:14.346201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14575 MB memory:  -> device: 1, name: Tesla V100-SXM2-16GB, pci bus id: 0000:06:00.0, compute capability: 7.0
2023-03-03 13:12:14.346759: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14575 MB memory:  -> device: 2, name: Tesla V100-SXM2-16GB, pci bus id: 0000:07:00.0, compute capability: 7.0
2023-03-03 13:12:14.347281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14575 MB memory:  -> device: 3, name: Tesla V100-SXM2-16GB, pci bus id: 0000:08:00.0, compute capability: 7.0



2023-03-03 13:12:17.150486: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype float and shape [60000,28,28,1]
	 [[{{node Placeholder/_0}}]]
2023-03-03 13:12:17.150814: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype uint8 and shape [60000]
	 [[{{node Placeholder/_1}}]]
INFO:tensorflow:batch_all_reduce: 8 all-reduces with algorithm = nccl, num_packs = 1
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:batch_all_reduce: 8 all-reduces with algorithm = nccl, num_packs = 1
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
2023-03-03 13:12:22.182639: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:424] Loaded cuDNN version 8800
2023-03-03 13:12:22.192166: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:424] Loaded cuDNN version 8800
2023-03-03 13:12:22.213909: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:424] Loaded cuDNN version 8800
2023-03-03 13:12:22.227182: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:424] Loaded cuDNN version 8800
2023-03-03 13:12:23.789883: I tensorflow/tsl/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
2023-03-03 13:12:24.937923: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at nccl_ops.cc:108 : UNKNOWN: Error invoking NCCL: unhandled cuda error
2023-03-03 13:12:24.937999: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at nccl_ops.cc:108 : UNKNOWN: Error invoking NCCL: unhandled cuda error
2023-03-03 13:12:24.938051: I tensorflow/core/common_runtime/executor.cc:1197] [/job:localhost/replica:0/task:0/device:GPU:1] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNKNOWN: Error invoking NCCL: unhandled cuda error
	 [[{{node NcclAllReduce_1}}]]
2023-03-03 13:12:24.938075: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at nccl_ops.cc:108 : UNKNOWN: Error invoking NCCL: unhandled cuda error
2023-03-03 13:12:24.938105: I tensorflow/core/common_runtime/executor.cc:1197] [/job:localhost/replica:0/task:0/device:GPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNKNOWN: Error invoking NCCL: unhandled cuda error
	 [[{{node NcclAllReduce}}]]
2023-03-03 13:12:24.938124: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at nccl_ops.cc:108 : UNKNOWN: Error invoking NCCL: unhandled cuda error
2023-03-03 13:12:24.938175: I tensorflow/core/common_runtime/executor.cc:1197] [/job:localhost/replica:0/task:0/device:GPU:3] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNKNOWN: Error invoking NCCL: unhandled cuda error
	 [[{{node NcclAllReduce_3}}]]
2023-03-03 13:12:24.938202: I tensorflow/core/common_runtime/executor.cc:1197] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNKNOWN: Error invoking NCCL: unhandled cuda error
	 [[{{node NcclAllReduce_1}}]]
	 [[Identity_5/_92]]
2023-03-03 13:12:24.938224: I tensorflow/core/common_runtime/executor.cc:1197] [/job:localhost/replica:0/task:0/device:GPU:2] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNKNOWN: Error invoking NCCL: unhandled cuda error
	 [[{{node NcclAllReduce_2}}]]
Output exceeds the size limit. Open the full output data in a text editor
---------------------------------------------------------------------------
UnknownError                              Traceback (most recent call last)
Cell In[14], line 18
     16 num_batches = 0
     17 for x in train_dist_dataset:
---> 18   total_loss += distributed_train_step(x)
     19   num_batches += 1
     20 train_loss = total_loss / num_batches

File /home/anaconda3/envs/tf_env/lib/python3.10/site-packages/tensorflow/python/util/traceback_utils.py:153, in filter_traceback..error_handler(*args, **kwargs)
    151 except Exception as e:
    152   filtered_tb = _process_traceback_frames(e.__traceback__)
--> 153   raise e.with_traceback(filtered_tb) from None
    154 finally:
    155   del filtered_tb

File /home/anaconda3/envs/tf_env/lib/python3.10/site-packages/tensorflow/python/eager/execute.py:52, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     50 try:
     51   ctx.ensure_initialized()
---> 52   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     53                                       inputs, attrs, num_outputs)
     54 except core._NotOkStatusException as e:
     55   if name is not None:

UnknownError: Graph execution error:

Detected at node 'NcclAllReduce' defined at (most recent call last):
    File "/home/anaconda3/envs/tf_env/lib/python3.10/runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/home/anaconda3/envs/tf_env/lib/python3.10/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel_launcher.py", line 17, in 
      app.launch_new_instance()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/traitlets/config/application.py", line 992, in launch_instance
      app.start()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 711, in start
      self.io_loop.start()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 215, in start
      self.asyncio_loop.run_forever()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
      self._run_once()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/asyncio/base_events.py", line 1906, in _run_once
      handle._run()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/asyncio/events.py", line 80, in _run
      self._context.run(self._callback, *self._args)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 510, in dispatch_queue
      await self.process_one()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 499, in process_one
      await dispatch(*args)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 406, in dispatch_shell
      await result
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 729, in execute_request
      reply_content = await reply_content
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 411, in do_execute
      res = shell.run_cell(
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 531, in run_cell
      return super().run_cell(*args, **kwargs)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 2961, in run_cell
      result = self._run_cell(
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3016, in _run_cell
      result = runner(coro)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
      coro.send(None)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3221, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3400, in run_ast_nodes
      if await self.run_code(code, result, async_=asy):
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3460, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "/tmp/ipykernel_16086/4166678551.py", line 18, in 
      total_loss += distributed_train_step(x)
    File "/tmp/ipykernel_16086/4166678551.py", line 5, in distributed_train_step
      per_replica_losses = strategy.run(train_step, args=(dataset_inputs,))
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/keras/optimizers/utils.py", line 175, in _all_reduce_sum_fn
      return distribution.extended.batch_reduce_to(
Node: 'NcclAllReduce'
Detected at node 'NcclAllReduce_3' defined at (most recent call last):
    File "/home/anaconda3/envs/tf_env/lib/python3.10/runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/home/anaconda3/envs/tf_env/lib/python3.10/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel_launcher.py", line 17, in 
      app.launch_new_instance()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/traitlets/config/application.py", line 992, in launch_instance
      app.start()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 711, in start
      self.io_loop.start()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 215, in start
      self.asyncio_loop.run_forever()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
      self._run_once()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/asyncio/base_events.py", line 1906, in _run_once
      handle._run()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/asyncio/events.py", line 80, in _run
      self._context.run(self._callback, *self._args)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 510, in dispatch_queue
      await self.process_one()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 499, in process_one
      await dispatch(*args)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 406, in dispatch_shell
      await result
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 729, in execute_request
      reply_content = await reply_content
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 411, in do_execute
      res = shell.run_cell(
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 531, in run_cell
      return super().run_cell(*args, **kwargs)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 2961, in run_cell
      result = self._run_cell(
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3016, in _run_cell
      result = runner(coro)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
      coro.send(None)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3221, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3400, in run_ast_nodes
      if await self.run_code(code, result, async_=asy):
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3460, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "/tmp/ipykernel_16086/4166678551.py", line 18, in 
      total_loss += distributed_train_step(x)
    File "/tmp/ipykernel_16086/4166678551.py", line 5, in distributed_train_step
      per_replica_losses = strategy.run(train_step, args=(dataset_inputs,))
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/keras/optimizers/utils.py", line 175, in _all_reduce_sum_fn
      return distribution.extended.batch_reduce_to(
Node: 'NcclAllReduce_3'
Detected at node 'NcclAllReduce_2' defined at (most recent call last):
    File "/home/anaconda3/envs/tf_env/lib/python3.10/runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/home/anaconda3/envs/tf_env/lib/python3.10/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel_launcher.py", line 17, in 
      app.launch_new_instance()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/traitlets/config/application.py", line 992, in launch_instance
      app.start()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 711, in start
      self.io_loop.start()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 215, in start
      self.asyncio_loop.run_forever()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
      self._run_once()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/asyncio/base_events.py", line 1906, in _run_once
      handle._run()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/asyncio/events.py", line 80, in _run
      self._context.run(self._callback, *self._args)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 510, in dispatch_queue
      await self.process_one()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 499, in process_one
      await dispatch(*args)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 406, in dispatch_shell
      await result
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 729, in execute_request
      reply_content = await reply_content
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 411, in do_execute
      res = shell.run_cell(
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 531, in run_cell
      return super().run_cell(*args, **kwargs)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 2961, in run_cell
      result = self._run_cell(
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3016, in _run_cell
      result = runner(coro)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
      coro.send(None)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3221, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3400, in run_ast_nodes
      if await self.run_code(code, result, async_=asy):
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3460, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "/tmp/ipykernel_16086/4166678551.py", line 18, in 
      total_loss += distributed_train_step(x)
    File "/tmp/ipykernel_16086/4166678551.py", line 5, in distributed_train_step
      per_replica_losses = strategy.run(train_step, args=(dataset_inputs,))
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/keras/optimizers/utils.py", line 175, in _all_reduce_sum_fn
      return distribution.extended.batch_reduce_to(
Node: 'NcclAllReduce_2'
Detected at node 'NcclAllReduce_1' defined at (most recent call last):
    File "/home/anaconda3/envs/tf_env/lib/python3.10/runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/home/anaconda3/envs/tf_env/lib/python3.10/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel_launcher.py", line 17, in 
      app.launch_new_instance()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/traitlets/config/application.py", line 992, in launch_instance
      app.start()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 711, in start
      self.io_loop.start()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 215, in start
      self.asyncio_loop.run_forever()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
      self._run_once()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/asyncio/base_events.py", line 1906, in _run_once
      handle._run()
    File "/home/anaconda3/envs/tf_env/lib/python3.10/asyncio/events.py", line 80, in _run
      self._context.run(self._callback, *self._args)
...
	 [[Identity_5/_92]]
  (4) UNKNOWN:  Error invoking NCCL: unhandled cuda error
	 [[{{node NcclAllReduce_1}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_distributed_train_step_4197]
```
</details>