"torch.utils.data.DataLoader"'s documentation for how to properly set
up random seeds in workers with its "worker_init_fn" option.
My recurrent network doesn't work with data parallelism
There is a subtlety in using the "pack sequence -> recurrent network
-> unpack sequence" pattern in a "Module" with "DataParallel" or
"data_parallel()". Input to each the "forward()" on each device will
only be part of the entire input. Because the unpack operation
"torch.nn.utils.rnn.pad_packed_sequence()" by default only pads up to
the longest input it sees, i.e., the longest on that particular
device, size mismatches will happen when results are gathered
together. Therefore, you can instead take advantage of the
"total_length" argument of "pad_packed_sequence()" to make sure that
the "forward()" calls return sequences of same length. For example,
you can write:
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence