<p dir="auto">Some of of cuDNN's algorithms are non-deterministic, even with the seed set to X, for example</p>
<div class="highlight highlight-source-c notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="typedef enum
{
    CUDNN_CONVOLUTION_BWD_FILTER_ALGO_0         = 0,  // non-deterministic
    CUDNN_CONVOLUTION_BWD_FILTER_ALGO_3         = 3,  // non-deterministic, algo0 with workspace"><pre class="notranslate"><span class="pl-k">typedef</span> <span class="pl-k">enum</span>
{
    <span class="pl-c1">CUDNN_CONVOLUTION_BWD_FILTER_ALGO_0</span>         <span class="pl-c1">=</span> <span class="pl-c1">0</span>,  <span class="pl-c">// non-deterministic</span>
    <span class="pl-c1">CUDNN_CONVOLUTION_BWD_FILTER_ALGO_3</span>         <span class="pl-c1">=</span> <span class="pl-c1">3</span>,  <span class="pl-c">// non-deterministic, algo0 with workspace</span></pre></div>
<div class="highlight highlight-source-c notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="typedef enum
{
    CUDNN_CONVOLUTION_BWD_DATA_ALGO_0                 = 0, // non-deterministic"><pre class="notranslate"><span class="pl-k">typedef</span> <span class="pl-k">enum</span>
{
    <span class="pl-c1">CUDNN_CONVOLUTION_BWD_DATA_ALGO_0</span>                 <span class="pl-c1">=</span> <span class="pl-c1">0</span>, <span class="pl-c">// non-deterministic</span></pre></div>
<p dir="auto">(This list is probably non-exhaustive)</p>
<p dir="auto">PyTorch currently provides no way for the user to select which specific algorithms to use in cuDNN. However, this is fine, since PyTorch was designed to be more high-level and abstract than some of the other frameworks.</p>
<p dir="auto">Perhaps instead of exposing the entire cuDNN algorithm choices, a boolean variable specifying non-determinism vs determinism should be added? This would be along the lines of the current <code class="notranslate">torch.backends.cudnn.enabled</code> option.</p>
<p dir="auto">Currently, users have to resort to CPU or disabling cuDNN for reproducibility. Some optimizers do work deterministically (at least for us) on the GPU (e.g. SGD, Adam), but Adadelta and RMSprop do not.</p>
<p dir="auto">It is totally possible to implement deterministic training with lua torch avoiding some cunn modules and cudnn algos. cutorch has atomicAdd in THIndexTensor which shouldn't pose a problem.<br>
PyTorch should implement:</p>
<ul dir="auto">
<li>ordered parallel data loading similar to torchnet <a href="https://github.com/torchnet/torchnet/blob/master/dataset/paralleldatasetiterator.lua#L53">https://github.com/torchnet/torchnet/blob/master/dataset/paralleldatasetiterator.lua#L53</a></li>
<li>cudnn should allow user set deterministic mode somehow</li>
</ul>