utils.py
def truncated_bptt_batch_generator(input_seqs, target_seqs, max_batch_size, max_truncation_length, num_epochs=np.inf,
                                   shuffle=True, random_roll=False):
  """ Generate batches for truncated backpropagation through time.

  All sequences in a batch will be cast to floats. To make sequence lengths
  the same for an individual batch, inputs are padded with 0.0 and targets
  are padded with nan.

  TODO: Make this more efficient. In particular, when sequence lengths vary
  dramatically within a dataset, we should (a) maintain `batch_size` different
  full-BPTT generators, (b) start a new sequence immediately after another is
  exhausted, and (c) maintain sequence-level resets instead of batch-level
  resets. This would also require a model that can handle such sequence-level
  resets.

  Args:
    input_seqs: A list of 2-D NumPy arrays, each with shape
      `[seq_length, input_size]`.
    target_seqs: A list of 2-D NumPy arrays, each with shape
      `[seq_length, target_size]`.
    max_batch_size: An integer. Smaller batch sizes are possible when
      `max_batch_size` doesn't divide the total number of sequences that we'll
      end up visiting in `num_epochs` epochs.
    max_truncation_length: An integer. Shorter lengths are possible when
      `max_truncation_length` doesn't evenly divide all sequence lengths.
    num_epochs: An integer or float. The number of epochs to visit.
    shuffle: A boolean. If True, shuffle sequences epoch by epoch.
    random_roll: A boolean. If True, randomly roll each input, target
      sequence along the time axis (of course before splitting the sequence
      up for truncated backprop).

  Yields:
    A tuple `(batch_inputs, batch_targets, reset_states)`. `batch_inputs` and
    `batch_targets` are both 3-D float NumPy arrays with shape
    `[batch_size, batch_length, size]`. `reset_states` is a boolean
    indicating whether states should be reset before processing the batch.
  """

  full_bptt_batch_gen = full_bptt_batch_generator(input_seqs, target_seqs, max_batch_size, num_epochs,
                                                  shuffle=shuffle, random_roll=random_roll)
  for batch_inputs, batch_targets in full_bptt_batch_gen:
    length = batch_inputs.shape[1]
    split_ind = list(range(max_truncation_length, length, max_truncation_length))
    split_batch_inputs = np.array_split(batch_inputs, split_ind, axis=1)
    split_batch_targets = np.array_split(batch_targets, split_ind, axis=1)
    for i, (batch_inputs, batch_targets) in enumerate(zip(split_batch_inputs, split_batch_targets)):
      yield batch_inputs, batch_targets, i == 0
 Generate batches for truncated backpropagation through time.

  All sequences in a batch will be cast to floats. To make sequence lengths
  the same for an individual batch, inputs are padded with 0.0 and targets
  are padded with nan.

  TODO: Make this more efficient. In particular, when sequence lengths vary
  dramatically within a dataset, we should (a) maintain `batch_size` different
  full-BPTT generators, (b) start a new sequence immediately after another is
  exhausted, and (c) maintain sequence-level resets instead of batch-level
  resets. This would also require a model that can handle such sequence-level
  resets.

  Args:
    input_seqs: A list of 2-D NumPy arrays, each with shape
      `[seq_length, input_size]`.
    target_seqs: A list of 2-D NumPy arrays, each with shape
      `[seq_length, target_size]`.
    max_batch_size: An integer. Smaller batch sizes are possible when
      `max_batch_size` doesn't divide the total number of sequences that we'll
      end up visiting in `num_epochs` epochs.
    max_truncation_length: An integer. Shorter lengths are possible when
      `max_truncation_length` doesn't evenly divide all sequence lengths.
    num_epochs: An integer or float. The number of epochs to visit.
    shuffle: A boolean. If True, shuffle sequences epoch by epoch.
    random_roll: A boolean. If True, randomly roll each input, target
      sequence along the time axis (of course before splitting the sequence
      up for truncated backprop).

  Yields:
    A tuple `(batch_inputs, batch_targets, reset_states)`. `batch_inputs` and
    `batch_targets` are both 3-D float NumPy arrays with shape
    `[batch_size, batch_length, size]`. `reset_states` is a boolean
    indicating whether states should be reset before processing the batch.
  
Generate batches for truncated backpropagation through time.
All sequences in a batch will be cast to floats. To make sequence lengths
the same for an individual batch, inputs are padded with 0.0 and targets
are padded with nan.

Make this more efficient. In particular, when sequence lengths vary
dramatically within a dataset, we should (a) maintain `batch_size` different
full-BPTT generators, (b) start a new sequence immediately after another is
exhausted, and (c) maintain sequence-level resets instead of batch-level
resets. This would also require a model that can handle such sequence-level
resets.
Generate batches for truncated backpropagation through time . All sequences in a batch will be cast to floats . To make sequence lengths the same for an individual batch inputs are padded with 0 . 0 and targets are padded with nan . Make this more efficient . In particular when sequence lengths vary dramatically within a dataset we should ( a ) maintain ` batch_size ` different full - BPTT generators ( b ) start a new sequence immediately after another is exhausted and ( c ) maintain sequence - level resets instead of batch - level resets . This would also require a model that can handle such sequence - level resets .
def truncated_bptt_batch_generator(input_seqs, target_seqs, max_batch_size, max_truncation_length, num_epochs=np.inf,
                                   shuffle=True, random_roll=False):
  full_bptt_batch_gen = full_bptt_batch_generator(input_seqs, target_seqs, max_batch_size, num_epochs,
                                                  shuffle=shuffle, random_roll=random_roll)
  for batch_inputs, batch_targets in full_bptt_batch_gen:
    length = batch_inputs.shape[1]
    split_ind = list(range(max_truncation_length, length, max_truncation_length))
    split_batch_inputs = np.array_split(batch_inputs, split_ind, axis=1)
    split_batch_targets = np.array_split(batch_targets, split_ind, axis=1)
    for i, (batch_inputs, batch_targets) in enumerate(zip(split_batch_inputs, split_batch_targets)):
      yield batch_inputs, batch_targets, i == 0
def truncated_bptt_batch_generator ( input_seqs , target_seqs , max_batch_size , max_truncation_length , num_epochs = np . inf , shuffle = True , random_roll = False ) : full_bptt_batch_gen = full_bptt_batch_generator ( input_seqs , target_seqs , max_batch_size , num_epochs , shuffle = shuffle , random_roll = random_roll ) for batch_inputs , batch_targets in full_bptt_batch_gen : length = batch_inputs . shape [ 1 ] split_ind = list ( range ( max_truncation_length , length , max_truncation_length ) ) split_batch_inputs = np . array_split ( batch_inputs , split_ind , axis = 1 ) split_batch_targets = np . array_split ( batch_targets , split_ind , axis = 1 ) for i , ( batch_inputs , batch_targets ) in enumerate ( zip ( split_batch_inputs , split_batch_targets ) ) : yield batch_inputs , batch_targets , i == 0
Generate batches for truncated backpropagation through time.
Generate batches for truncated backpropagation through time .
""" Generate batches for truncated backpropagation through time.

  All sequences in a batch will be cast to floats. To make sequence lengths
  the same for an individual batch, inputs are padded with 0.0 and targets
  are padded with nan.

  TODO: Make this more efficient. In particular, when sequence lengths vary
  dramatically within a dataset, we should (a) maintain `batch_size` different
  full-BPTT generators, (b) start a new sequence immediately after another is
  exhausted, and (c) maintain sequence-level resets instead of batch-level
  resets. This would also require a model that can handle such sequence-level
  resets.

  Args:
    input_seqs: A list of 2-D NumPy arrays, each with shape
      `[seq_length, input_size]`.
    target_seqs: A list of 2-D NumPy arrays, each with shape
      `[seq_length, target_size]`.
    max_batch_size: An integer. Smaller batch sizes are possible when
      `max_batch_size` doesn't divide the total number of sequences that we'll
      end up visiting in `num_epochs` epochs.
    max_truncation_length: An integer. Shorter lengths are possible when
      `max_truncation_length` doesn't evenly divide all sequence lengths.
    num_epochs: An integer or float. The number of epochs to visit.
    shuffle: A boolean. If True, shuffle sequences epoch by epoch.
    random_roll: A boolean. If True, randomly roll each input, target
      sequence along the time axis (of course before splitting the sequence
      up for truncated backprop).

  Yields:
    A tuple `(batch_inputs, batch_targets, reset_states)`. `batch_inputs` and
    `batch_targets` are both 3-D float NumPy arrays with shape
    `[batch_size, batch_length, size]`. `reset_states` is a boolean
    indicating whether states should be reset before processing the batch.
  """
A tuple `(batch_inputs, batch_targets, reset_states)`. `batch_inputs` and
`batch_targets` are both 3-D float NumPy arrays with shape
`[batch_size, batch_length, size]`. `reset_states` is a boolean
indicating whether states should be reset before processing the batch.
A tuple ` ( batch_inputs batch_targets reset_states ) ` . ` batch_inputs ` and ` batch_targets ` are both 3 - D float NumPy arrays with shape ` [ batch_size batch_length size ] ` . ` reset_states ` is a boolean indicating whether states should be reset before processing the batch .
input_seqs
A list of 2-D NumPy arrays, each with shape
`[seq_length, input_size]`.
A list of 2 - D NumPy arrays each with shape ` [ seq_length input_size ] ` .
target_seqs
A list of 2-D NumPy arrays, each with shape
`[seq_length, target_size]`.
A list of 2 - D NumPy arrays each with shape ` [ seq_length target_size ] ` .
max_batch_size
An integer. Smaller batch sizes are possible when
`max_batch_size` doesn't divide the total number of sequences that we'll
end up visiting in `num_epochs` epochs.
An integer . Smaller batch sizes are possible when ` max_batch_size ` doesn ' t divide the total number of sequences that we ' ll end up visiting in ` num_epochs ` epochs .
max_truncation_length
An integer. Shorter lengths are possible when
`max_truncation_length` doesn't evenly divide all sequence lengths.
An integer . Shorter lengths are possible when ` max_truncation_length ` doesn ' t evenly divide all sequence lengths .
num_epochs
An integer or float. The number of epochs to visit.
An integer or float . The number of epochs to visit .
shuffle
A boolean. If True, shuffle sequences epoch by epoch.
A boolean . If True shuffle sequences epoch by epoch .
random_roll
A boolean. If True, randomly roll each input, target
sequence along the time axis (of course before splitting the sequence
up for truncated backprop).
A boolean . If True randomly roll each input target sequence along the time axis ( of course before splitting the sequence up for truncated backprop ) .