Does it matter if I bake a Normal map from a model withouth Color?
If you're not going to add bump, displacement or normal textures onto the high poly model, then it does not make a difference wether you bake it before or after adding textures.

All sculpted detail will properly be baked regardless of (color) textures.