Displace modifier does not match the texture
Ok everything you saw in my first post was properly set, except that my unwrapping was wrong : some parts of the mesh (a complex one btw) were crossing each other on the UV map, causing the wrong displacement mapping.

That was pretty obvious in the end.

I deliberately chose to cross some parts of it because I wanted a specific texture mapping. So, for any person interested in this post, I guess that it would be possible to correctly map the displacement with some crossing unwrapped parts of the mesh by using multiple UV map.