<p dir="auto">Currently cuDNN-based RNNs in TF are limited to GradientDescentOptimizer (<a href="https://github.com/tensorflow/tensorflow/issues/6620" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/6620/hovercard">#6620</a>). This is a serious limitation given the widespread use of other optimizers. The claim that this cannot be supported in TF because cuDNN RNN don't have known shapes at static time seems overly pessimistic. Just like RNNParamsSaver provides a mechanism to convert between canonical shaped variables and the parameter buffer, a simple wrapper can be provided that does this automatically for cuDNN RNNs, so that the size of the parameter buffer is statically calculated and then used to define the variable, especially since the shape of the parameter buffer is just a 1D vector anyway.</p>
<p dir="auto">I am testing how to use CudnnLSTM, there is not a lot of documentation on this. In my own experiment, I found when use AdamOptimizer with CudnnLSTM, it always raises the following Exception.</p>
<p dir="auto">I also found another repository using CudnnLSTM, and uploaded it here: <a href="https://github.com/boche/LM-PTB-CUDNNLSTM">https://github.com/boche/LM-PTB-CUDNNLSTM</a>. It also raises the same exception.</p>
<blockquote>
<p dir="auto">File "ptb_word_lm.py", line 465, in main<br>
m = PTBModel(is_training=True, config=config, debug=FLAGS.debug)<br>
File "ptb_word_lm.py", line 254, in <strong>init</strong><br>
self._train_op = optimizer.apply_gradients(zip(allgrads, allvars))<br>
File "/data/ASR1/ramons/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 409, in apply_gradients<br>
self._create_slots(var_list)<br>
File "/data/ASR1/ramons/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/training/adam.py", line 119, in _create_slots<br>
self._zeros_slot(v, "m", self._name)<br>
File "/data/ASR1/ramons/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 609, in _zeros_slot<br>
named_slots[var] = slot_creator.create_zeros_slot(var, op_name)<br>
File "/data/ASR1/ramons/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/training/slot_creator.py", line 121, in create_zeros_slot<br>
val = array_ops.zeros(primary.get_shape().as_list(), dtype=dtype)<br>
File "/data/ASR1/ramons/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/tensor_shape.py", line 782, in as_list<br>
raise ValueError("as_list() is not defined on an unknown TensorShape.")<br>
ValueError: as_list() is not defined on an unknown TensorShape.</p>
</blockquote>
<p dir="auto">For now, using GradientDescentOptimizer is okay, but it seems all other fancy optimizers all have similar problems. I have looked up this problem on stackoverflow, and found a related thread: <a href="http://stackoverflow.com/questions/40698821/tensorflow-adamoptimizer-throws-error-when-variable-has-validate-shape-false" rel="nofollow">http://stackoverflow.com/questions/40698821/tensorflow-adamoptimizer-throws-error-when-variable-has-validate-shape-false</a> .</p>
<p dir="auto">It seems the problem is that parameter buffer used in CudnnLSTM doesn't have fixed shape (validate_shape=False, defined in line 145 of <a href="https://github.com/boche/LM-PTB-CUDNNLSTM/blob/master/ptb_word_lm.py">https://github.com/boche/LM-PTB-CUDNNLSTM/blob/master/ptb_word_lm.py</a> ) , which is required by AdamOptimizer.</p>
<p dir="auto">Cudnn seems to be faster (less time per epoch), but if we can't use better learning algorithms with it (meaning more epochs), then the total running time may not be improved so much.</p>