Is the dielectric layer under the top or bottom copper layers always prepreg or can it be a core?
It can be done either way, but conventionally the surface layers are copper foil over prepreg.

Here are two different ways to do a 6-layer board:



The one on the right is the standard stackup style of every board house I've used.