How to blend World Space Normals
Huge thanks to @MJP who answered this.
The aim is to avoid the simplification made when using tangent space normals.
Here is the paper : Blending in detail

But only implement equation (4) which gives you this.

float3 ReorientNormal(in float3 u, in float3 t, in float3 s)
{
    // Build the shortest-arc quaternion
    float4 q = float4(cross(s, t), dot(s, t) + 1) / sqrt(2 * (dot(s, t) + 1));
    // Rotate the normal
    return u * (q.w * q.w - dot(q.xyz, q.xyz)) + 2 * q.xyz * dot(q.xyz, u) + 2 * q.w * cross(q.xyz, u);
}


Original answer is here.