Using 3-channel (RGB) PyTorch model for classification 4-channel (RGBY) images
I have labeled dataset with 4-channel images (RGBY). I want to use pretrained classification model (using pytorch and ResNet50 as a model). All of pytorch models for 3 channels though.
So, the question is: How can I use 3-channel pretrained models for 4-channels data? I'm loading the model next way:
import torchvision.models as models
resnet50 = models.resnet50(pretrained=True)


You can modify the first layer of the CNN such that it expects 4 input channels instead of 3. In your case, the first layer is resnet50.conv1. So:
import torchvision.models as models
resnet50 = models.resnet50(pretrained=True)

# modify first layer so it expects 4 input channels; all other parameters unchanged
resnet50.conv1 = torch.nn.Conv2d(4,64,kernel_size = (7,7),stride = (2,2), padding = (3,3), bias = False) 

# test
inp = torch.rand([1,4,512,512])
resnet50.eval()
resnet50.training = False
out = resnet50(inp) # should evaluate without error

The simplicity of this change is made possible by the following implementation detail: For a 2D convolution (also true for other dimensional convolutions), pytorch convolves one kernel for each desired output plane (feature map) with each input plane. This results in n_input_planes x n_output_planes total feature maps (in this case 4 and 64, respectively). Pytorch then sums across all input planes for each output plane, yielding a total of n_output_planes planes regardless of the number of input planes.
The good news is that this means you can add additional input planes (maps) with no modification of the network past the first layer. The (perhaps in some cases) unfavorable part of this is that all of your input feature maps are treated identically and the information from each is fully incorporated by the end of  the first convolution. In some cases it might be desirable to treat the input feature maps differently at the beginning, in which case you'd need to define two separate CNN branches so that the features were not added together at each layer.
