Comparison between addition and multiplication function in deep neural network? I designed a specific Convolution Neural Network to study in the area of image processing. The network has a part that there are two tensors which have to be transformed into a tensor in order to be fed to the next layer. This situation happen in several points of the network. In fact, there are several operations such as addition, multiplication, etc. The results of the network are a bit better when I  use the addition pyramid pooling module (the second image between two convolution) and multiply function (in last step of the network). I used tf.math.add and tf.math.multiply which perform the operation element-wisely. The whole network is shown in the first image.



The second image represents the pyramid pooling module which includes several scale images. 



I am looking forward to the addition and multiplication function's attribute in a deep neural network. 

The question is:

Why does the addition function (between conv1 and conv2) indicate better final performance in Accuracy (precision) and mean Intersection of Union(mIoU) compared to multiplication and concatenation when I unify two tensors into one tensor?
The observation is very interesting you report, since concatenation and addition are practically the same. A nice explanation can be found in https://distill.pub/2018/feature-wise-transformations/ .