Baked and applied normals reflect light at an angle
Found it. I had to add a normal map between the wood displacement EXR and the normals.
Simply because I didn't know this was the way to allow blender to interpret image to normals as described here

Actually, the person who deleted his answer to my question deserves all the credit. So thanks :)