Rendered Normals color getting distorted?
I think the main cause of the differences is the way data is encoded when the normals were rendered initially.
If you save that in jpg or png, colors are encoded as integers. So that leads to some rounded values that cannot be converted to the exact initial normal values.
So you can save this rendered file with OpenExr format to keep things as float values:

(codec can be &quot;none&quot; or &quot;zip looseless&quot; for accuracy, if needed... give a try to the codecs depending on what you want, file size vs. accuracy).
The result can be this (not perfect though):

Instead of this using png:

Additionally, keep the image texture as &quot;non color&quot; or &quot;raw&quot;, and as you said in the question, set &quot;raw&quot; and &quot;non color&quot; in the color management part.
Edit: other thing that alter the rendering is anti aliasing. If using Cycles, set the pixel filter to box in the film section. Rendered back using the image texture projected on the window should give a better result.