Merge by distance messes up shading?
So the problem wasn't in the "Merge by distance", but rather because after I imported the model all edges were marked as sharp and I cleared them. If I don't clear them everything works fine. So to anyone having the same problem leave the edges marked as sharp.