Normal/Bump map show in Look Dev but not on Render
Okay, I did some more testing around and finally came to a solution. Apparently a Mix Shader into a Shader to RGB make the bump map get renderer in Look Dev but not in Render. The solution (and in hindsight the more reasonable) approach is to use a Mix RGB node instead.
I'm not sure if this can be considerer as just plain stupidity or of it's an actual bug.

Here is my new setup that works: