How to change ferrite core of transformer
The biggest thing to (usually) worry about is core saturation. Core saturation occurs for ferrites from somewhere between 200 mT and 400 mT and this can be gradual or more pronounced and, is dependent on the ferrite material in the core. So, it's the quantity of milli teslas (mT) that you need to concern yourself with and, this relates to the applied peak current through the inductor using this formula: -

$$\mu H = B$$


\$\mu\$ is the magnetic permeability of the core material and equals \$\mu_0\cdot\mu_r\$ where \$\mu_0\$ is the baseline magnetic permeability of air/vacuum (\$1.26E-6\$) and \$\mu_r\$ is the relative permeability of the ferrite material used (somewhere between 1 and several thousand).
B is the flux density (maximum limit between 200 mT and 400 mT)
H is the ampere turns applied (magneto motive force) divided by the mean length around the core. Ampere-turns are what they are i.e. peak current x number of turns.


So if the core generally has a shorter mean length, the H field gets bigger and the core is likely to saturate more: Mean length around the core: -



The distance you are interested in is \$\ell_e\$ shown in the picture above and, for more complex core shapes the value of \$\ell_e\$ will be in the data sheet.

However, if you choose a smaller ferrite core, it might have less inductance for the number of turns and you might need to increase the new inductance to keep things on a level playing field. This means increasing the number of turns. But, it is likely that your peak current will need to remain the same so instantly, two things happen: -


A smaller ferrite leads to a natural increase in H field (\$\ell_e\$ shortening)
A smaller ferrite needs more turns and hence, for the same operating peak current, the magneto motive force increases and therefore so does the H field.


In other words, choosing a smaller core has to be done carefully because there are dual mechanisms at play that can easily cause unwanted core saturation levels. Even just down-sizing by one core-size can lead to a significant worsening in performance but, it all depends how close the original core was sailing towards saturation.

You also need to ensure that the core is suitable for the operating frequency of the switching circuit so, go with a core material that is the same as used previously (or better). The core material data sheets can help with this.

Increasing (or adding) the (an) air gap (at the expense of winding more turns to restore inductance) can usually lead to satisfactory performance for smaller cores but, beware of copper loss with the extra turns.