Setting a normal map for only part of a material (single Principled shader)
It's probably easier to use separate Principled BSDF nodes for the wood and metal, or have the "clean" region over the metal in the normal map itself. (depends on how you're painting this normal map). 

However, if you want to use the setup in your example, it can work. You just need to use the geometry node to set the alternate normal to the default mesh normal instead of 1 (which is what the unconnected white socket does):