RGB normals not summing up rightly.
They should have a magnitude of (approximately) 1. I say approximately as precision could be an issue. 

You need to make sure that you are mapping the rgb values correctly back into decimal values, each colour range should map to the decimal range -1 to 1 with 128 mapping to 0. 

You mention in comments that you renormalise by dividing by 255, this is incorrect. You should divide by 128 and subtract 1.

Note that the z / blue channel has its sign flipped compared to the others, but this shouldn't be an issue checking the magnitude. Also some software also flips other channels.

See https://en.wikipedia.org/wiki/Normal_mapping#How_it_works