How to calculate the maximum output current of a flyback design?
I just want to calculate the maximum output current


Max output current is wholly related to maximum output power and boost regulators (single inductor) and flyback regulators (transformer) are virtually the same analytically. The cycle of switching is split into two halves: -


Store energy in the inductor (boost) or primary (flyback)
Release energy into the output (load and capacitor)


For a simple discontinuous analysis (inductor or primary current allowed to deplete to zero amps), the peak current (prior to release) represents energy stored. If this is dumped into the output then there is so-much energy transferred every switching cycle. 

This means that power output is energy per cycle stored (and released) multiplied by switching frequency. This ignores losses but should give you a general feel.

Energy is stored in the inductor or primary inductance and, during that storing process, nothing is transferred to the secondary due to the way the dots on the transformer are positioned. So, for a flyback design, the transformer primary attains a peak current and the energy stored is: -

\$\dfrac{LI^2}{2}\$

If you want more power you have to attain a higher value of primary current and, because current rises linearly (for a given input voltage rail) you have to have a longer MOSFET on duty. This is where transformer saturation problems can happen so you have to be careful about this.


  More specifically, I want to know how can you calculate the max output
  current of this particular design


There isn't enough information available about the transformer to help you on this and, noting that you don't want someone else to do your homework, I expect you can now work through this. You have to: -


determine the primary inductance
determine the peak current
look at details on the transformer core (material and shape)
Figure out the H field (maximum)
Estimate the B field
See if the core isn't saturating too much