<p dir="auto">Please:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Check for duplicate requests.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Describe your goal, and if possible provide a code snippet with a motivating example.</li>
</ul>
<h3 dir="auto">Description</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2023-05-29 05:39:19.492073: E external/xla/xla/stream_executor/cuda/cuda_dnn.cc:427] Loaded runtime CuDNN library: 8.5.0 but source was compiled with: 8.6.0.  CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
Traceback (most recent call last):
  File &quot;/t5x_home/t5x/train.py&quot;, line 834, in &lt;module&gt;
    config_utils.run(main)
  File &quot;/t5x_home/t5x/config_utils.py&quot;, line 214, in run
    gin_utils.run(main)
  File &quot;/t5x_home/t5x/gin_utils.py&quot;, line 129, in run
    app.run(
  File &quot;/usr/local/lib/python3.8/dist-packages/absl/app.py&quot;, line 312, in run
    _run_main(main, args)
  File &quot;/usr/local/lib/python3.8/dist-packages/absl/app.py&quot;, line 258, in _run_main
    sys.exit(main(argv))
  File &quot;/t5x_home/t5x/train.py&quot;, line 787, in main
    _main(argv)
  File &quot;/t5x_home/t5x/train.py&quot;, line 829, in _main
    train_using_gin()
  File &quot;/usr/local/lib/python3.8/dist-packages/gin/config.py&quot;, line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File &quot;/usr/local/lib/python3.8/dist-packages/gin/utils.py&quot;, line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File &quot;/usr/local/lib/python3.8/dist-packages/gin/config.py&quot;, line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File &quot;/t5x_home/t5x/train.py&quot;, line 265, in train
    rng = random.PRNGKey(random_seed)
  File &quot;/usr/local/lib/python3.8/dist-packages/jax/_src/random.py&quot;, line 137, in PRNGKey
    key = prng.seed_with_impl(impl, seed)
  File &quot;/usr/local/lib/python3.8/dist-packages/jax/_src/prng.py&quot;, line 320, in seed_with_impl
    return random_seed(seed, impl=impl)
  File &quot;/usr/local/lib/python3.8/dist-packages/jax/_src/prng.py&quot;, line 734, in random_seed
    return random_seed_p.bind(seeds_arr, impl=impl)
  File &quot;/usr/local/lib/python3.8/dist-packages/jax/_src/core.py&quot;, line 380, in bind
    return self.bind_with_trace(find_top_trace(args), args, params)
  File &quot;/usr/local/lib/python3.8/dist-packages/jax/_src/core.py&quot;, line 383, in bind_with_trace
    out = trace.process_primitive(self, map(trace.full_raise, args), params)
  File &quot;/usr/local/lib/python3.8/dist-packages/jax/_src/core.py&quot;, line 790, in process_primitive
    return primitive.impl(*tracers, **params)
  File &quot;/usr/local/lib/python3.8/dist-packages/jax/_src/prng.py&quot;, line 746, in random_seed_impl
    base_arr = random_seed_impl_base(seeds, impl=impl)
  File &quot;/usr/local/lib/python3.8/dist-packages/jax/_src/prng.py&quot;, line 751, in random_seed_impl_base
    return seed(seeds)
  File &quot;/usr/local/lib/python3.8/dist-packages/jax/_src/prng.py&quot;, line 1419, in _rbg_seed
    halfkey = threefry_seed(seed)
  File &quot;/usr/local/lib/python3.8/dist-packages/jax/_src/prng.py&quot;, line 980, in threefry_seed
    return _threefry_seed(seed)
jaxlib.xla_extension.XlaRuntimeError: FAILED_PRECONDITION: DNN library initialization failed. Look at the errors above for more details.
  In call to configurable 'train' (&lt;function train at 0x7efc24d265e0&gt;)
"><pre class="notranslate"><code class="notranslate">2023-05-29 05:39:19.492073: E external/xla/xla/stream_executor/cuda/cuda_dnn.cc:427] Loaded runtime CuDNN library: 8.5.0 but source was compiled with: 8.6.0.  CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
Traceback (most recent call last):
  File "/t5x_home/t5x/train.py", line 834, in &lt;module&gt;
    config_utils.run(main)
  File "/t5x_home/t5x/config_utils.py", line 214, in run
    gin_utils.run(main)
  File "/t5x_home/t5x/gin_utils.py", line 129, in run
    app.run(
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "/t5x_home/t5x/train.py", line 787, in main
    _main(argv)
  File "/t5x_home/t5x/train.py", line 829, in _main
    train_using_gin()
  File "/usr/local/lib/python3.8/dist-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/usr/local/lib/python3.8/dist-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/usr/local/lib/python3.8/dist-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/t5x_home/t5x/train.py", line 265, in train
    rng = random.PRNGKey(random_seed)
  File "/usr/local/lib/python3.8/dist-packages/jax/_src/random.py", line 137, in PRNGKey
    key = prng.seed_with_impl(impl, seed)
  File "/usr/local/lib/python3.8/dist-packages/jax/_src/prng.py", line 320, in seed_with_impl
    return random_seed(seed, impl=impl)
  File "/usr/local/lib/python3.8/dist-packages/jax/_src/prng.py", line 734, in random_seed
    return random_seed_p.bind(seeds_arr, impl=impl)
  File "/usr/local/lib/python3.8/dist-packages/jax/_src/core.py", line 380, in bind
    return self.bind_with_trace(find_top_trace(args), args, params)
  File "/usr/local/lib/python3.8/dist-packages/jax/_src/core.py", line 383, in bind_with_trace
    out = trace.process_primitive(self, map(trace.full_raise, args), params)
  File "/usr/local/lib/python3.8/dist-packages/jax/_src/core.py", line 790, in process_primitive
    return primitive.impl(*tracers, **params)
  File "/usr/local/lib/python3.8/dist-packages/jax/_src/prng.py", line 746, in random_seed_impl
    base_arr = random_seed_impl_base(seeds, impl=impl)
  File "/usr/local/lib/python3.8/dist-packages/jax/_src/prng.py", line 751, in random_seed_impl_base
    return seed(seeds)
  File "/usr/local/lib/python3.8/dist-packages/jax/_src/prng.py", line 1419, in _rbg_seed
    halfkey = threefry_seed(seed)
  File "/usr/local/lib/python3.8/dist-packages/jax/_src/prng.py", line 980, in threefry_seed
    return _threefry_seed(seed)
jaxlib.xla_extension.XlaRuntimeError: FAILED_PRECONDITION: DNN library initialization failed. Look at the errors above for more details.
  In call to configurable 'train' (&lt;function train at 0x7efc24d265e0&gt;)

</code></pre></div>
<h3 dir="auto">What jax/jaxlib version are you using?</h3>
<p dir="auto">jax 0.4.10  jaxlib-0.4.10+cuda11.cudnn86-cp38-cp38</p>
<h3 dir="auto">Which accelerator(s) are you using?</h3>
<p dir="auto">GPU</p>
<h3 dir="auto">Additional system info</h3>
<p dir="auto">ubuntu 22.04<br>
cuda 11.8<br>
cudnn 8.8</p>
<h3 dir="auto">NVIDIA GPU info</h3>
<p dir="auto"><a target="_blank" rel="noopener noreferrer" href="https://user-images.githubusercontent.com/24863775/241653973-c51e5cfc-4390-425d-9cf6-c47aaa283c3e.png"><img src="https://user-images.githubusercontent.com/24863775/241653973-c51e5cfc-4390-425d-9cf6-c47aaa283c3e.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto"><em>No response</em></p>