Move an object after baking normals
I've found the issue and I feel stupid now but maybe someone will feel like me one day so here is what happened : Before I bake normals, I add a modifier on my low poly (shrinkwrap) and I forgot to shut it down... :p