Multiple-input multiple-output CNN with custom loss function I have a set of 2D input arrays $(n\times m)$ namely $A,B,C$ and I have to predict two 2D output arrays namely $d,e$ for which I do have the expected values. You can think of the inputs/outputs as grey images if you like.

Because of the spatial information is relevant (there are actually 2D physical domains) I want to use a Convolutional Neural Network to predict $d$ and $e$. My design (not tested yet) looks as follows:



Because I have multiple inputs, I guess I should use multiple columns (or branches) to find different features for each of the inputs (they look fairly different). Each of these columns follows a encoding-decoding architecture used in segmentation (see SegNet): Conv2D block involves a convolution+batch normalisation+ReLU layer. Deconv2D involves a deconvolution+batch normalisation+ReLU.

Then, I can merge the output of each column by either concatenating, averaging or taking the maximum for example. To obtain the original $m\times n$ shape for each of the outputs I have seen I could do this with a $1\times1$ kernel convolution.

I want to predict the two outputs from that single layer. Is that okay from the network structure point of view? Finally my loss function depends on the outputs themselves compared to the target plus another relation I want to impose.

A would like to have some expert opinion on this since this is my first design of a CNN and I am not sure if I it makes sense as it is now and/or if there are better approaches (or network architectures) to this problem.
Looks correct. When u use same branches in NN architecture - this name siamese-type nn. U may see there how to implement it in PyPorch. In my opinion, PyTorch must easiest framework for this task. Also implementation of encoder-decoder network u can find there.

This is frequently using technique when u need to compute, as example, distance between few inputs. But i'm had doubts about concatenation part. Witch dimension u need for outputs? If same as input, then all looks fine. If different dim, then just add fully connected layers after last convolution.