Self adjusting CNN network
Spatial pyramid pooling layers (https://arxiv.org/pdf/1406.4729.pdf) should solve this problem for you. These layers allow you to use input images of any dimension, instead of being restricted to 224x224 images, for example.