Is convenient/needed to relocate while asbestos is removed from external panles in my flat?
The risk is someone could inhale asbestos fibers and develop health problems far into the future. It's impossible to estimate the level of this risk, yet you need to do so in order to decide whether to vacate or not. Furthermore, if fibers could enter your flat, even if you vacated during the work, the fibers could still remain upon your return.

If the contractor doing the removal work is fully qualified to do this sort of work, the risk to your family should be negligible, yet there is always some risk even if infinitely small. Only you can decide if you're really comfortable with this or not, and what an appropriate response is. This is of course difficult without hard data, but there just isn't any.

One thing you could do is some airborne monitoring. This is often done as part of removal procedures anyway. I've no idea of the costs, and it only reflects levels during the testing period, but it may buy you some peace of mind.