can the efficiency of flyback converter be less than 50%?
On practice the efficiency can be "as low as it wishes to be".
Under 50% is far from uncommon at the ends of various ranges.
eg many converters which may achieve 80% - 90% or even better at optimum operating point often have efficiencies in the 20%-40% range at eg very low load levels where losses do not reduce proportional to power level.  For example, an output diode may have a say 0.7V mean voltage drop  at 1A and say 0.5V at 100 mA and not much less at 10 mA. At low output voltages this loss alone may represent a significant efficiency loss.

If formulae are based on certain assumptions and the results are significantly different than the assumptions then if accuracy is important it is wise to examine the initial derivations/formulae/assumptions to see if the results will change significantly if the assumptions are redone.  In some cases the result will be minimal change, in others it may be a make-or-break difference.

A simple example of an assumption that may change is the duty cycle of a flyback converter and the effect on eg peak input and output current . In an ideal converter Ton/Toff = Vin/Vout. For efficiency Z  (Z = %_efficiency / 100) &lt; 1 the on time % will increase to provide more energy to make up for the losses and the mean input current will rise (as Vin x Iin_mean = (Iout x Iout_mean)/Z ). The  output energy must be transferred in less time (if frequency does not change) so output current pulse amplitudes must increase. At very low efficiencies the output diode may be driven above its peak repetitive current rating (although mean current will be unchanged) and input switch will have increased mean current and maybe peak current. 
If semiconductors were close to maximum specified value when efficiency formulae were derived they may be driven beyond specified limits in much lower efficiency cases.