How can I create convolutions or linear layers that operate on vectors rather than scalars in pytorch?
If I understand your question1, you'd like an input tensor of shape (2, 3), where each element is actually a vector. If that is the case, you are essentially talking about a 3d tensor (instead of 2d). 

For example, if each vector is 5 elements long, your input would have the shape (2, 3, 5).

To compute convolutions over this, there are layers that take the dimensions as parameters - have a look at the Convolutional layers like Conv3d.

To compute a linear layer on this input, you still just need to flatten or reshape the tensor to be a single vector.

If you vector lengths can vary, you will have to consider making them all match by either truncating them (making some shorter), or padding (adding e.g. zeros to make shorter ones longer) - or a mixture of both techniques.



1 If I haven't fully understood, please add more details to the example in your question