Create a tensor view (no data copy) where one row/col is removed
TL;DR: Is there a way to delete one row or column of a torch tensor's view without creating a copy of the underlying data?
Background:
Starting from a base tensor (samples, features) of shape (2000,10000) (about 90MB), I want to create a 2-tuple of new views for each feature that consist of the one-feature-training-target with shape (2000,1) and the training inputs with shape (2000,9999) containing the remaining features.
As I want to train one rather simple model for each feature (10k) and as many as possible in parallel, I would like to have this data available in parallel, but my memory does not allow for this unless the input tensors are views.
While slicing creates views and can easily be used for the targets, so far I haven't found a slicing or function that can be used for the inputs. Creating two slices and concatenating them again creates a copy. The same happens when I use a mask to remove the unwanted feature.

Currently, this is not possible.

PyTorch's dense tensors are always represented by a dense data array and the view only changes where the indices break into the next dimension.

PyTorch's sparse tensors only do the opposite of what you intend, they allow to spread the densely stored data to be interpreted as being spread out.


That being said, you could design your models to take the (2000, 10000) sized tensor as input and effectively remove the target input-feature from the output of your first layer.
Let's say your first layer is a torch.nn.Linear which means that there is an underlying matrix of shape (10000, d_out) (where d_out is the intended output size of the layer - actually, the current implementation uses the transposed weight matrix but I'll ignore that in the examples unless it's explicitly layer.weight that's being used). So your first internal feature tensor is mathematically equivalent to:
layer_output = x.matmul(weights) + biases

which you can rid of input feature i_target by:
layer_output = x.matmul(weights) + biases - x[:,i_target].unsqueeze(dim=-1) * weights[i_target]

or for layer = torch.nn.Linear(10000, d_out):
layer_output = layer(x) - x[:,i_target].unsqueeze(dim=-1) * layer.weight[:,i_target]

So the rest of the network needs no adjustments. A similar approach is also possible for convolution layers.
Also note that this is only necessary for training. If you want your final model to take samples with 9999 features you can just erase the i_target-th column from the linear layer's weights matrix and remove the
- x[:,i_target].unsqueeze(dim=-1) * weights[i_target]

part.
Final note:
Personally I wouldn't train 10k individual models but rather have a single model with 10k outputs where each of the outputs only depends on 9999 of the input features. Of course this makes the model design much more difficult as it requires a lot of care to make sure that each layer's output contains features that don't depend on one of the input features, but it removes a lot of communication overhead that comes with processing lots of small operations on the GPU (which can be devastating in my experience).
