Multilayer PCB - are there voids in substrate between layers?
typically, multi-layer boards consist of single-sided or double sided PCBs glued together. So that glue/lacquer fills these voids – the typical few µm aren't a problem to fill.

Now, I haven't thought about that, but obviously, a board manufacturer would choose a glue/lacquer that has comparable dielectric properties as your substrate. For FR4, that might be quite possible; whether that works with more exotic materials such as ceramic substrates would be something I'd definitely have to ask.