High Voltage Planar Transformers
We need a lot more information to give you good advice. For example, how high of a voltage makes a huge difference on the size limitation, what frequency are you thinking about, whatever info you can share makes this easier. The numbers can turn this from difficult to impossible very easily.

The main thing to be worried about is straight up breakdown, you need to cover the clearance distance to for the high voltage you are working with, for example you make a 6kV DC/DC with air insulation you will need at least 2cms for it not to breakdown instantly when the voltage is applied(you will of course want more than this distance). You can of course use different materials

You essentially consider insulators as capacitors, and frequency affects them quite a bit so what frequency you work with also has an effect on this element.

The other main thing you have to worry about is your transformer saturating, this will kill your output on the DC end and even if you clear the distance the converter will be useless. So first you choose what material your transformer will be and see if the size needed for that transformer to work suits your needs

you mainly look at the BH curve, some makers do not provide them so it can be hard to design sometimes.

you mainly look at these expressions:

$$
B=\frac{E}{4.44NAf}
$$

where B is magnetic field E is rms voltage applied(not the high voltage internally at what voltage you are working with), N is number of turns A is area of your core and finally the frequency.

$$
H=\frac{IdcN}{Le}
$$

Idc is the DC current, you can use rms if you are working with sine waves or other stuff, N is number of turns and Le is effective magnetic path length, you will have to look up how to measure this.

then you can go to a ferrite catalog like this one

look at the frequency curves, make sure it even works at your frequency and make sure it is not on the elbows of the curve(you must have an idea of what are your input and output waveform while you check this)

Even if you have a ferrite material that actually can do the job, then you must check that it can also handle the high voltage you are applying by itself. In this case high voltage breakdown will always be your highest priority.