Neural style transfer loss fucntion
So why we do not use gram matrix in loss function calculation between content and generated?



"Same content" means same objects in same locations in the image. If the original content shows a face at the top, we want to see a stylised face at the top.
"Same style" means same combinations of textures and line style together in any location in the image. The gram matrix is a good proxy measure of this, because it is invariant to the locations at which the style correlations occur, and similar line shapes and textures will tend to trigger the same combinations of low-level feature map channels.


If you used gram matrix for the higher order content channels, you would probably find that larger objects would be treated like a style and would be moved, duplicated and distorted in the final output. Depending on the trained network, which layer(s) you picked as representing content etc, the effect might still be interesting or artistic. But it would no longer be style transfer, but perhaps some hybrid of style transfer and deep dream.