saturation level of Kool Mu material
You have to design to all limits, thermal as well as magnetic.

When deciding the maximum flux density to use, the frequency is important. A simple model of what's going on is that a core loses a certain amount of energy as it cycles once round its hysteresis loop. In this model, core dissipation is proportional to frequency. 

In practice, the faster the core is taken round the loop, the higher the energy lost, so the power dissipation is proportional to a power higher than 1 of the frequency. Unfortunately, the KoolMu loss data is only given at 100kHz, and gives no hint of how it changes with frequency. In general, if something is not specified, you can expect it to be worse than you think rather than better, so this lack should be a warning that perhaps 100kHz is the highest useful frequency you can use.

The frequency is important in a flyback, as it works entirely by stored energy, so power throughput (assuming constant stored energy, which is of course not true) rises with frequency as well.

If you intend to use 100kHz or less for the frequency, then you can be reasonably safe in assuming that the core power loss will fall at least as fast as linearly with frequency, and so can increase your projected flux density to get a tolerable power loss at your chosen frequency. Once you have reached such a low frequency that you are permitted more than 1T, then you are limited by saturation.