How to create a 1 to 1 feed forward layer?
I'm familiar with a fully connected layer, but how can I create a custom layer in PyTorch that is just 1 to 1? That is, each neuron is only connected to 1 other neuron.
Example: Layer 1 neurons: a,b,c
Layer 2 neurons d,e,f
Connections:
a-d
b-e
c-f

Linear layers are basically just describing a matrix multiplication. And since this is not what you want you can't use the Pytorch implementation nn.Linear. You want to have each weight correspond to just one input neuron and one output neuron. That would mean that the amount of output neurons must be the same as the amount of input neurons (which limits the possibilities of a neural network very much, are you sure you want that?).
So it is basically just elementwise multiplication where one factor is learnable.
An example of what this kind if linear layer could look like:
class ElementwiseLinear(nn.Module):
    def __init__(self, input_size: int) -&gt; None:
        super(ElementwiseLinear, self).__init__()

        # w is the learnable weight of this layer module
        self.w = nn.Parameter(torch.rand(input_size), requires_grad=True)

    def forward(self, x: torch.tensor) -&gt; torch.tensor:
        # simple elementwise multiplication
        return self.w * x

Now an example of how to use it in a model:
class Model(nn.Module):
    def __init__(self, input_size: int) -&gt; None:
        super(Model, self).__init__()
        
        self.elementWiselinear1 = ElementwiseLinear(input_size)
        self.elementWiselinear2 = ElementwiseLinear(input_size)
        self.elementWiselinear3 = ElementwiseLinear(input_size)

    def forward(self, x: torch.tensor) -&gt; torch.tensor:
        x = F.relu(self.elementWiselinear1(x))
        x = F.relu(self.elementWiselinear2(x))
        x = torch.sigmoid(self.elementWiselinear3(x))

        return x

Again, I dont know what this could be useful for but I hope thats what you wanted!
