PCB Stack-up Window in Altium on the plane layer
What I generally do, is just place the numbers on plane layers, and create cuts in the copper along the bar/window in all the other (plane) layers.

That way you get a transparent square with a solid number for each signal layer and a solid square with a transparent number for each plane layer, allowing you to also see if all layers were processed correctly.

Although these days I also often create small electrical structures to verify alignment and presence of all layers -if this is required at all-, as that takes much less space to verify than something you need to be able to distinguish through 1.6mm of some modern materials, with the exception of X-raying or such practices.