Multi-image superresolution using CNNs I'm trying to write a program that can take multiple low-resolution images as inputs and output a high-resolution image.

My understanding is that for single-image superresolution, Convolutional Neural Networks work great. I can just take a network with just three convolution layers as described here. I can then train the network with a huge dataset of low-res images and their corresponding high-res images (with mean-squared error cost function), and it should theoretically work. 

If I want to achieve greater accuracy, though, I can gain more information about a potential high-resolution image by looking at multiple low-resolution images. 

So my question is: How would I modify this neural-network based algorithm to be able to take multiple images (of the same object) as input, and output a more accurate high-resolution image as a result? I would assume this is a lot more complicated, because the multiple images might be off by a fraction of a pixel, or taken from a marginally different angle.

(Sorry if anything I said is incorrect; I know virtually nothing about this field, so any advice helps!)
I would assume this is a lot more complicated, because the multiple images might be off by a fraction of a pixel, or taken from a marginally different angle.


I agree with your analysis that this problem presents different challenges than single-image super-resolution.

In the single image case, we know exactly where everything should be in the output image, because it is just a higher resolution version of the input.

So why not try and extend this idea to your case? Here is how you might do that:


Concatenate each low-res image and a low-res version of the output image as each training and test example
For prediction concatenate each low-res image and duplicate one of them to specify the desired orientation of the output


So if you have n 3-channel h x w input images in each training example, your input shape should be (h, w, 3*(n+1))

This should give the network enough information to learn the correct transformation.