Conversion of magnitudes to Jansky and MAGPHYS?
Caveat this with the fact that I am not familiar with this code.

The input asks you for the flux through the filter, but a filter has a bandpass, so if you have a 10 nm bandpass filter and you measure $1 W/m^2$, then you measured $0.1 W/m^2/nm$ (sorry, I'm a physicist so I never got used to measuring wavelengths in Hz).

As for your second question, the factor of 23 is because in cgs units a Jansky is $10^{23} erg/s/Hz/cm^{-1}$ and that TOPCAT equation is in cgs (the equation is $m_\text{AB} = -\frac{5}{2}\,log_{10}\,f_v - 48.6$) and if you are normalizing it to Janskys, you need to divide by $10^{-23}$.