How to calculate air gap in flyback transformer?
Power output and energy per switching cycle

An output of 265 volts at 5 mA is a power of 1.325 watts and this means that the energy that needs to be transfered each switching cycle is 1.325 W divided by the switching frequency. Hence, the energy released by the flyback transformer is 2.65 µJ. Accounting for losses, you should probably bump that up to around 3.3 µJ.



Using a primary inductance of 56 µH

3.3 µJ is the amount of energy needed in each switching cycle so, if you assume DCM operation and a conservative maximum duty cycle of 75% (1.5 µs) we can say: -


Energy stored per cycle (W) is 3.3 µJ because DCM uses all the energy stored.
This requires a peak primary current of \$\sqrt{\dfrac{2\cdot W}{L}}\$ = 343 mA
The lowest \$V_{SUPPLY}\$ that achieves this is \$L\cdot\dfrac{dI}{dt}\$ = \$56 µH\cdot\dfrac{0.343A}{1.5 µs}\$ = 12.8 volts.


So, immediately I'm not all that confident that your choice of 56 µH is good. It's a little close to not working at the lower voltage supply using my assumptions. Yes, we might believe that a slightly lower energy per cycle (say more like 3 µJ) would be fine and, that would require a primary current of 327 mA. But, the minimum supply rail would still be be 12.2 volts. Or, we could also make the maximum duty cycle closer to 90% (1.8 µs) and that would allow a supply voltage as low as 10.2 volts.



Using a primary inductance of 26 µH

But personally, I'd go for lowering the inductance because I also know that winding many hundreds of turns for the secondary is a pain and if you can get away with fewer turns, all the better. So, I'm going to go for 26 µH instead of 56 µH. We can now say: -


This requires a peak primary current of \$\sqrt{\dfrac{2\cdot W}{L}}\$ = 504 mA
The lowest \$V_{SUPPLY}\$ that achieves this is \$L\cdot\dfrac{dI}{dt}\$ = \$26 µH\cdot\dfrac{0.504A}{1.5 µs}\$ = 8.74 volts.


So far so good - with your minimum supply of 12 volts, this should be fairly easy.



Number of primary turns

The ungapped core-set in the question has an \$A_L\$ value of 1200 so, to achieve 26 µH needs circa 5 turns (30 µH). 



H-field calculation

The core-set has a mean effective length (\$\ell_e\$) of 47 mm hence, we can now say what the peak H-field will be: -


H-field is 0.504 A x 5 turns divided by 0.047 metres = 53.6 At/m.




B-field calculation

Using \$\mu_r\$ in the data sheet, that H-field will produce a peak flux density of: -


\$1450 \times 4\pi \times 10^{-7}\times 53.6\$ = 98 mT




To gap or not to gap


  I have problem with air gap: I know that I need one, but I have no
  clue how much gap I need


You should avoid peak flux densities much over 200 mT so I don't think you need one. However, if you have got your current output wrong and meant to say 50 mA then you will likely need one but, the same data sheet gives options: -



In red is the ungapped core set values used above. The row directly above gives \$A_L\$ at 315 and \$\mu_r\$ of 380 for a gap of 0.12 mm for example. To get 26 µH requires 9 turns (25.5 µH) etc..

If you need further help leave a comment.



Formulas used

Inductor energy (W) equation: -
$$W = \dfrac{L\cdot I^2}{2} \Longrightarrow I = \sqrt{\dfrac{2\cdot W}{L}}$$