Neural network layer without all connections
The weights in a dense layer of a neural network is a (n,d) matrix, and I want to force some of these weights to always be zero.  I have another (n,d) matrix which is the mask of which entries can be non-zero.  The idea is that the layer should not be truly dense, but have some connections missing (i.e. equal to 0).
How can achieve this while training with PyTorch (or Tensorflow)?  I don't want these weights to become non-zero while training.
One method, if it doesn't support it directly, would be to zero-out the desired entries after each iteration of training.

You can take advantage of pytorch's sparse data type:
class SparseLinear(nn.Module):
  def __init__(self, in_features, out_features, sparse_indices):
    super(SparseLinear, self).__init__()
    self.weight = nn.Parameter(data=torch.sparse.FloatTensor(sparse_indices, torch.randn(sparse_indices.shape[1]), [in_features, out_features]), requires_grad=True)
    self.bias = nn.Parameter(data=torch.randn(out_features), requires_grad=True)

  def forward(self, x):
    return torch.sparse.admm(self.bias, self.weight, x, 1., 1.)


