Is there a many to many convolution in Pytorch? is this a thing?
I have been thinking about convolutions recently. There are common 3by3 convs, where (3,3) kernel's information is weighted and aggregated to supply information to a single spatial point on the output. There are also 3 by 3 upconvs, where a single spatial point on the input supplies weighted information to a 3 by 3 output space.
The conv is a many to one relationship and the upconv is a one to many relationship.
I have however never heard of many to many conv? is there such a thing? For example, a 3by3 kernel supplying information to another 3by3 kernel. I would like to experiment with it in PyTorch. My internet searching has not revealed anything.

You can combine pixel shuffle and averaging to get what you want.
for example, if you want 3x3 -&gt; 3x3 mapping with in_channels to out_channels:
from torch import nn
import torch.nn.functional as nnf


class ManyToManyConv2d(nn.Module):
  def __init__(in_channels, out_channels, in_kernel, out_kernel):
    self.out_kernel = out_kernel
    self.conv = nn.Conv2d(in_channels, out_channles * out_kernel * out_kernel, in_kernel)

  def forward(self, x):
    y = self.conv(x)  # all the output kernel are &quot;folded&quot; into the channel dim
    y = nnf.pixel_shuffle(y, self.out_kernel)  # &quot;unfold&quot; the out_kernel - image size *out_kernel bigger
    y = nnf.avg_pool2d(y, self.out_kernel)
    return y

