<p dir="auto">With this issue I would like to track the efforts in integrating the<br>
cudnn library within tensorflow.</p>
<p dir="auto">As of June 17th 2016 doing a manual grep on the repository gives these<br>
functions as being mapped from cudnn to the stream executor,</p>
<p dir="auto">From chapter 4 of the cudnn User Guide version 5.0 (April 2016):</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnGetVersion</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetErrorString</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnCreate</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnDestroy</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnSetStream</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetStream</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnCreateTensorDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetTensor4dDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetTensor4dDescriptorEx</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetTensor4dDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnSetTensorNdDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnDestroyTensorDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnTransformTensor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnAddTensor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnOpTensor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnSetTensor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnScaleTensor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnCreateFilterDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetFilter4dDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetFilter4dDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetFilter4dDescriptor_v3 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetFilter4dDescriptor_v3 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetFilter4dDescriptor_v4 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetFilter4dDescriptor_v4 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnSetFilterNdDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetFilterNdDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetFilterNdDescriptor_v3 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetFilterNdDescriptor_v3 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetFilterNdDescriptor_v4 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetFilterNdDescriptor_v4 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnDestroyFilterDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnCreateConvolutionDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetConvolution2dDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetConvolution2dDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetConvolution2dForwardOutputDim</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnSetConvolutionNdDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetConvolutionNdDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnGetConvolutionNdForwardOutputDim</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnDestroyConvolutionDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnFindConvolutionForwardAlgorithm</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnFindConvolutionForwardAlgorithmEx</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnGetConvolutionForwardAlgorithm</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnGetConvolutionForwardWorkspaceSize</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnConvolutionForward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnConvolutionBackwardBias</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnFindConvolutionBackwardFilterAlgorithm</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnFindConvolutionBackwardFilterAlgorithmEx</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnGetConvolutionBackwardFilterAlgorithm</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnGetConvolutionBackwardFilterWorkspaceSize</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnConvolutionBackwardFilter</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnFindConvolutionBackwardDataAlgorithm</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnFindConvolutionBackwardDataAlgorithmEx</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnGetConvolutionBackwardDataAlgorithm</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnGetConvolutionBackwardDataWorkspaceSize</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnConvolutionBackwardData</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSoftmaxForward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSoftmaxBackward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnCreatePoolingDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetPooling2dDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetPooling2dDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnSetPoolingNdDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetPoolingNdDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetPooling2dDescriptor_v3 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetPooling2dDescriptor_v3 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetPoolingNdDescriptor_v3 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetPoolingNdDescriptor_v3 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetPooling2dDescriptor_v4 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetPooling2dDescriptor_v4 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetPoolingNdDescriptor_v4 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetPoolingNdDescriptor_v4 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnDestroyPoolingDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetPooling2dForwardOutputDim</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetPoolingNdForwardOutputDim</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnPoolingForward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnPoolingBackward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnActivationForward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnActivationBackward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnCreateActivationDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnSetActivationDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnGetActivationDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cudnnDestroyActivationDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnActivationForward_v3  (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnActivationBackward_v3 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnActivationForward_v4  (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnActivationBackward_v4 (versioned)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnCreateLRNDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetLRNDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetLRNDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnDestroyLRNDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnLRNCrossChannelForward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnLRNCrossChannelBackward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnDivisiveNormalizationForward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnDivisiveNormalizationBackward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnBatchNormalizationForwardInference</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnBatchNormalizationForwardTraining</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnBatchNormalizationBackward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnDeriveBNTensorDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnCreateRNNDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnDestroyRNNDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetRNNDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetRNNWorkspaceSize</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetRNNTrainingReserveSize</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetRNNParamsSize</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetRNNLinLayerMatrixParams</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnGetRNNLinLayerBiasParams</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnRNNForwardInference</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnRNNForwardTraining</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnRNNBackwardData</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnRNNBackwardWeights</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnCreateDropoutDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnDestroyDropoutDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnDropoutGetStatesSize</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnDropoutGetReserveSpaceSize</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetDropoutDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnDropoutForward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnDropoutBackward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnCreateSpatialTransformerDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnDestroySpatialTransformerDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSetSpatialTransformerNdDescriptor</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSpatialTfGridGeneratorForward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSpatialTfGridGeneratorBackward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSpatialTfSamplerForward</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> cudnnSpatialTfSamplerBackward</li>
</ul>
<h3 dir="auto">Batch Normalization</h3>
<p dir="auto">Seems <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/lukemetz/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/lukemetz">@lukemetz</a> was working on it but has stalled for a bit <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="145555764" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/1759" data-hovercard-type="pull_request" data-hovercard-url="/tensorflow/tensorflow/pull/1759/hovercard" href="https://github.com/tensorflow/tensorflow/pull/1759">#1759</a></p>
<h3 dir="auto">What is the plan for the RNN ?</h3>
<p dir="auto">I know <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/wchan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/wchan">@wchan</a> was working on a <a href="https://github.com/tensorflow/tensorflow/pull/2002" data-hovercard-type="pull_request" data-hovercard-url="/tensorflow/tensorflow/pull/2002/hovercard">cpu version</a>. I was trying to get a stub at <a href="https://github.com/Mistobaan/tensorflow/tree/feature/cudnn-rnn-lstm">using the cudnn version</a> but from the comments in that thread seemed like <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zheng-xq/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zheng-xq">@zheng-xq</a> was working on it internally.<br>
Can anyone comment on the status of these ops?</p>
<h3 dir="auto">Other Questions</h3>
<ol dir="auto">
<li>Any reasons why the Softmax functions are not being used ?</li>
<li>Would make sense to split the above list in chunks and create issues with contribution welcome, so external contributors can tackle them without duplicating internal work, or mark what google will be working on internally ?</li>
</ol>
<p dir="auto">I hope this helps in organizing the work around the cudnn and inspire the community to contribute. I will try to keep this issue up to date.</p>