class_docstr: lasagne.layers.recurrent.CustomRecurrentLayer(incoming, input_to_hidden,
  hidden_to_hidden, nonlinearity=lasagne.nonlinearities.rectify,
  hid_init=lasagne.init.Constant(0.), backwards=False,
  learn_init=False, gradient_steps=-1, grad_clipping=0,
  unroll_scan=False, precompute_input=True, mask_input=None,
  only_return_final=False, **kwargs)
  
  A layer which implements a recurrent connection.
  
  This layer allows you to specify custom input-to-hidden and
  hidden-to-hidden connections by instantiating :class:`lasagne.layers.Layer`
  instances and passing them on initialization.  Note that these connections
  can consist of multiple layers chained together.  The output shape for the
  provided input-to-hidden and hidden-to-hidden connections must be the same.
  If you are looking for a standard, densely-connected recurrent layer,
  please see :class:`RecurrentLayer`.  The output is computed by
  
  .. math ::
      h_t = \sigma(f_i(x_t) + f_h(h_{t-1}))
  
  Parameters
  ----------
  incoming : a :class:`lasagne.layers.Layer` instance or a tuple
      The layer feeding into this layer, or the expected input shape.
  input_to_hidden : :class:`lasagne.layers.Layer`
      :class:`lasagne.layers.Layer` instance which connects input to the
      hidden state (:math:`f_i`).  This layer may be connected to a chain of
      layers, which must end in a :class:`lasagne.layers.InputLayer` with the
      same input shape as `incoming`, except for the first dimension: When
      ``precompute_input == True`` (the default), it must be
      ``incoming.output_shape[0]*incoming.output_shape[1]`` or ``None``; when
      ``precompute_input == False``, it must be ``incoming.output_shape[0]``
      or ``None``.
  hidden_to_hidden : :class:`lasagne.layers.Layer`
      Layer which connects the previous hidden state to the new state
      (:math:`f_h`).  This layer may be connected to a chain of layers, which
      must end in a :class:`lasagne.layers.InputLayer` with the same input
      shape as `hidden_to_hidden`'s output shape.
  nonlinearity : callable or None
      Nonlinearity to apply when computing new state (:math:`\sigma`). If
      None is provided, no nonlinearity will be applied.
  hid_init : callable, np.ndarray, theano.shared or :class:`Layer`
      Initializer for initial hidden state (:math:`h_0`).
  backwards : bool
      If True, process the sequence backwards and then reverse the
      output again such that the output from the layer is always
      from :math:`x_1` to :math:`x_n`.
  learn_init : bool
      If True, initial hidden values are learned.
  gradient_steps : int
      Number of timesteps to include in the backpropagated gradient.
      If -1, backpropagate through the entire sequence.
  grad_clipping : float
      If nonzero, the gradient messages are clipped to the given value during
      the backward pass.  See [1]_ (p. 6) for further explanation.
  unroll_scan : bool
      If True the recursion is unrolled instead of using scan. For some
      graphs this gives a significant speed up but it might also consume
      more memory. When `unroll_scan` is True, backpropagation always
      includes the full sequence, so `gradient_steps` must be set to -1 and
      the input sequence length must be known at compile time (i.e., cannot
      be given as None).
  precompute_input : bool
      If True, precompute input_to_hid before iterating through
      the sequence. This can result in a speedup at the expense of
      an increase in memory usage.
  mask_input : :class:`lasagne.layers.Layer`
      Layer which allows for a sequence mask to be input, for when sequences
      are of variable length.  Default `None`, which means no mask will be
      supplied (i.e. all sequences are of the same length).
  only_return_final : bool
      If True, only return the final sequential output (e.g. for tasks where
      a single target value for the entire sequence is desired).  In this
      case, Theano makes an optimization which saves memory.
  
  Examples
  --------
  
  The following example constructs a simple `CustomRecurrentLayer` which
  has dense input-to-hidden and hidden-to-hidden connections.
  
  >>> import lasagne
  >>> n_batch, n_steps, n_in = (2, 3, 4)
  >>> n_hid = 5
  >>> l_in = lasagne.layers.InputLayer((n_batch, n_steps, n_in))
  >>> l_in_hid = lasagne.layers.DenseLayer(
  ...     lasagne.layers.InputLayer((None, n_in)), n_hid)
  >>> l_hid_hid = lasagne.layers.DenseLayer(
  ...     lasagne.layers.InputLayer((None, n_hid)), n_hid)
  >>> l_rec = lasagne.layers.CustomRecurrentLayer(l_in, l_in_hid, l_hid_hid)
  
  The CustomRecurrentLayer can also support "convolutional recurrence", as is
  demonstrated below.
  
  >>> n_batch, n_steps, n_channels, width, height = (2, 3, 4, 5, 6)
  >>> n_out_filters = 7
  >>> filter_shape = (3, 3)
  >>> l_in = lasagne.layers.InputLayer(
  ...     (n_batch, n_steps, n_channels, width, height))
  >>> l_in_to_hid = lasagne.layers.Conv2DLayer(
  ...     lasagne.layers.InputLayer((None, n_channels, width, height)),
  ...     n_out_filters, filter_shape, pad='same')
  >>> l_hid_to_hid = lasagne.layers.Conv2DLayer(
  ...     lasagne.layers.InputLayer(l_in_to_hid.output_shape),
  ...     n_out_filters, filter_shape, pad='same')
  >>> l_rec = lasagne.layers.CustomRecurrentLayer(
  ...     l_in, l_in_to_hid, l_hid_to_hid)
  
  References
  ----------
  .. [1] Graves, Alex: "Generating sequences with recurrent neural networks."
         arXiv preprint arXiv:1308.0850 (2013).
docstr: Compute this layer's output function given a symbolic input variable.
  
  Parameters
  ----------
  inputs : list of theano.TensorType
      `inputs[0]` should always be the symbolic input variable.  When
      this layer has a mask input (i.e. was instantiated with
      `mask_input != None`, indicating that the lengths of sequences in
      each batch vary), `inputs` should have length 2, where `inputs[1]`
      is the `mask`.  The `mask` should be supplied as a Theano variable
      denoting whether each time step in each sequence in the batch is
      part of the sequence or not.  `mask` should be a matrix of shape
      ``(n_batch, n_time_steps)`` where ``mask[i, j] = 1`` when ``j <=
      (length of sequence i)`` and ``mask[i, j] = 0`` when ``j > (length
      of sequence i)``. When the hidden state of this layer is to be
      pre-filled (i.e. was set to a :class:`Layer` instance) `inputs`
      should have length at least 2, and `inputs[-1]` is the hidden state
      to prefill with.
  
  Returns
  -------
  layer_output : theano.TensorType
      Symbolic output variable.
docstr_tok: ["Compute","this","layer","'s","output","function","given","a","symbolic","input","variable",".","Parameters","--","--","--","--","--","inputs",":","list","of","theano.TensorType","`","inputs","[","0","]","`","should","always","be","the","symbolic","input","variable",".","When","this","layer","has","a","mask","input","(","i.e",".","was","instantiated","with","`","mask_input","!","=","None","`",",","indicating","that","the","lengths","of","sequences","in","each","batch","vary",")",",","`","inputs","`","should","have","length","2",",","where","`","inputs","[","1","]","`","is","the","`","mask","`",".","The","`","mask","`","should","be","supplied","as","a","Theano","variable","denoting","whether","each","time","step","in","each","sequence","in","the","batch","is","part","of","the","sequence","or","not",".","`","mask","`","should","be","a","matrix","of","shape","``","(","n_batch",",","n_time_steps",")","``","where","``","mask","[","i",",","j","]","=","1","``","when","``","j","<","=","(","length","of","sequence","i",")","``","and","``","mask","[","i",",","j","]","=","0","``","when","``","j",">","(","length","of","sequence","i",")","``",".","When","the","hidden","state","of","this","layer","is","to","be","pre-filled","(","i.e",".","was","set","to","a",":","class",":","`","Layer","`","instance",")","`","inputs","`","should","have","length","at","least","2",",","and","`","inputs","[","-1","]","`","is","the","hidden","state","to","prefill","with",".","Returns","--","--","--","-","layer_output",":","theano.TensorType","Symbolic","output","variable","."]
code: def get_output_for(self, inputs, **kwargs):
      """
      Compute this layer's output function given a symbolic input variable.
  
      Parameters
      ----------
      inputs : list of theano.TensorType
          `inputs[0]` should always be the symbolic input variable.  When
          this layer has a mask input (i.e. was instantiated with
          `mask_input != None`, indicating that the lengths of sequences in
          each batch vary), `inputs` should have length 2, where `inputs[1]`
          is the `mask`.  The `mask` should be supplied as a Theano variable
          denoting whether each time step in each sequence in the batch is
          part of the sequence or not.  `mask` should be a matrix of shape
          ``(n_batch, n_time_steps)`` where ``mask[i, j] = 1`` when ``j <=
          (length of sequence i)`` and ``mask[i, j] = 0`` when ``j > (length
          of sequence i)``. When the hidden state of this layer is to be
          pre-filled (i.e. was set to a :class:`Layer` instance) `inputs`
          should have length at least 2, and `inputs[-1]` is the hidden state
          to prefill with.
  
      Returns
      -------
      layer_output : theano.TensorType
          Symbolic output variable.
      """
      # Retrieve the layer input
      input = inputs[0]
      # Retrieve the mask when it is supplied
      mask = None
      hid_init = None
      if self.mask_incoming_index > 0:
          mask = inputs[self.mask_incoming_index]
      if self.hid_init_incoming_index > 0:
          hid_init = inputs[self.hid_init_incoming_index]
  
      # Input should be provided as (n_batch, n_time_steps, n_features)
      # but scan requires the iterable dimension to be first
      # So, we need to dimshuffle to (n_time_steps, n_batch, n_features)
      input = input.dimshuffle(1, 0, *range(2, input.ndim))
      seq_len, num_batch = input.shape[0], input.shape[1]
  
      if self.precompute_input:
          # Because the input is given for all time steps, we can precompute
          # the inputs to hidden before scanning. First we need to reshape
          # from (seq_len, batch_size, trailing dimensions...) to
          # (seq_len*batch_size, trailing dimensions...)
          # This strange use of a generator in a tuple was because
          # input.shape[2:] was raising a Theano error
          trailing_dims = tuple(
              input.shape[n] for n in range(2, input.ndim)
          )
          input = T.reshape(
              input, (seq_len * num_batch,) + trailing_dims
          )
          input = helper.get_output(
              self.input_to_hidden, input, **kwargs
          )
  
          # Reshape back to (seq_len, batch_size, trailing dimensions...)
          trailing_dims = tuple(
              input.shape[n] for n in range(1, input.ndim)
          )
          input = T.reshape(input, (seq_len, num_batch) + trailing_dims)
  
      # We will always pass the hidden-to-hidden layer params to step
      non_seqs = helper.get_all_params(self.hidden_to_hidden)
      # When we are not precomputing the input, we also need to pass the
      # input-to-hidden parameters to step
      if not self.precompute_input:
          non_seqs += helper.get_all_params(self.input_to_hidden)
  
      # Create single recurrent computation step function
      def step(input_n, hid_previous, *args):
          # Compute the hidden-to-hidden activation
          hid_pre = helper.get_output(
              self.hidden_to_hidden, hid_previous, **kwargs
          )
  
          # If the dot product is precomputed then add it, otherwise
          # calculate the input_to_hidden values and add them
          if self.precompute_input:
              hid_pre += input_n
          else:
              hid_pre += helper.get_output(
                  self.input_to_hidden, input_n, **kwargs
              )
  
          # Clip gradients
          if self.grad_clipping:
              hid_pre = theano.gradient.grad_clip(
                  hid_pre, -self.grad_clipping, self.grad_clipping
              )
  
          return self.nonlinearity(hid_pre)
  
      def step_masked(input_n, mask_n, hid_previous, *args):
          # Skip over any input with mask 0 by copying the previous
          # hidden state; proceed normally for any input with mask 1.
          hid = step(input_n, hid_previous, *args)
          hid_out = T.switch(mask_n, hid, hid_previous)
          return [hid_out]
  
      if mask is not None:
          mask = mask.dimshuffle(1, 0, "x")
          sequences = [input, mask]
          step_fun = step_masked
      else:
          sequences = input
          step_fun = step
  
      if not isinstance(self.hid_init, Layer):
          # The code below simply repeats self.hid_init num_batch times in
          # its first dimension.  Turns out using a dot product and a
          # dimshuffle is faster than T.repeat.
          dot_dims = list(range(1, self.hid_init.ndim - 1)) + [
              0,
              self.hid_init.ndim - 1,
          ]
          hid_init = T.dot(
              T.ones((num_batch, 1)), self.hid_init.dimshuffle(dot_dims)
          )
  
      if self.unroll_scan:
          # Retrieve the dimensionality of the incoming layer
          input_shape = self.input_shapes[0]
          # Explicitly unroll the recurrence instead of using scan
          hid_out = unroll_scan(
              fn=step_fun,
              sequences=sequences,
              outputs_info=[hid_init],
              go_backwards=self.backwards,
              non_sequences=non_seqs,
              n_steps=input_shape[1],
          )[0]
      else:
          # Scan op iterates over first dimension of input and repeatedly
          # applies the step function
          hid_out = theano.scan(
              fn=step_fun,
              sequences=sequences,
              go_backwards=self.backwards,
              outputs_info=[hid_init],
              non_sequences=non_seqs,
              truncate_gradient=self.gradient_steps,
              strict=True,
          )[0]
  
      # When it is requested that we only return the final sequence step,
      # we need to slice it out immediately after scan is applied
      if self.only_return_final:
          hid_out = hid_out[-1]
      else:
          # dimshuffle back to (n_batch, n_time_steps, n_features))
          hid_out = hid_out.dimshuffle(1, 0, *range(2, hid_out.ndim))
  
          # if scan is backward reverse the output
          if self.backwards:
              hid_out = hid_out[:, ::-1]
  
      return hid_out
  
imports: import numpy
  import theano
  import theano.tensor
  from ..None import nonlinearities
  from ..None import init
  from ..utils import unroll_scan
  from .base import MergeLayer, Layer
  from .input import InputLayer
  from .dense import DenseLayer
  from .None import helper
label_desc: Use image node_id 4 for calling the CustomRecurrentLayer obj's underlying member method code with example usage: obj.get_output_for(inputs) and returns: hid_out, self, list