How to train pytorch model using batches with partial information
In this PyTorch model I have two streams data with two different modalities that are input into the model at the same time. The streams of data are blocks of sequential data. So I have modality one M1 = [[1,1,1,1],[2,2,2,2],[3,3,3,3],[4,4,4,4],[5,5,5,5]] and modality two M2 = [[11,11,11,11],[22,22,22,22],[33,33,33,33],[44,44,44,44]]. I want to train this model with a system that during training, batches of sequential data will randomly have partial or full information.
So there will be three possibilities during training: M1 and M2 will have its full sequential data, or M1 will be clipped meaning that the sequential data information will be set to zero (e.g. M1 = [[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0]] ), while M2 will have its full sequential data info, or M2 will be clipped meaning that the sequential data information in M2 will be set to zero, while M1 will have its full sequential data info.
Are there any PyTorch functions that will do that automatically for me, or does anyone know what would be a good way to implement this?

Let M1 and M2 be 2D tensors or 3D batched tensors for training, and let p1,p2 and p3 be the probabilities that M1, M2, or neither is zeroed:
p1 = 0.5 # for example
p2 = 0.3 # for example
# p3 = 1- p1 - p2 = 0.2

randn = torch.rand(1)

if randn &lt; p1:
   M1 *= 0
elif randn &gt; p1 and randn &lt; p1+p2:
   M2 *= 0

# pass M1 and M2 to your model

As a secondary note, if possible you may want to consider zeroing the gradient for whichever modality had its data erased. You don't really want your network to learn that the zero-values have any significance, which you are at risk of learning.
Depending on the structure of your model, this may be possible. For instance, if M1 is processed by 1 network branch and M2 is processed by a different branch, you could constrain so that loss is only back-propagated through the branch that received non-zeroed inputs. For other network structures though, this may not be possible.
