How do I bake this correctly?
Ok, tell me if i'm wrong (believe me I'd love to be wrong on this) but it turns out what I was trying to do is not possible because the diffuse texture is flat, and even when faking the normals, it still looks unrealistic. But it works if you bake them separately with just diffuse and color.