Cost function for LTI system identification
The chosen cost function is the mean squared error, i.e., the integral over a squared magnitude of the difference between frequency responses. The function

$$E(e^{j\omega})=H(e^{j\omega})-\frac{B(e^{j\omega})}{A(e^{j \omega})}\tag{1}$$

depends on frequency, so you can't minimize it directly, unless you want to minimize it for exactly one frequency $\omega$, which is of course pointless. You can choose several error measures depending on $E(e^{j\omega})$ given in $(1)$. Two common choices are

$$\varepsilon_1=\max_{\omega}W(\omega)|E(e^{j\omega})|\tag{2}$$

and

$$\varepsilon_2=\int_{0}^{\pi}W(\omega)|E(e^{j\omega})|^2d\omega\tag{3}$$

with some positive weighting function $W(\omega)$. Note that $\varepsilon_1$ and $\varepsilon_2$ given by $(2)$ and $(3)$ do not depend on frequency.

The authors of the paper you refer to chose to minimize the weighted mean square error given by $(3)$. However, instead of using the linear difference $(1)$, they chose to minimize the average logarithmic difference, i.e., the average error on a dB scale.

The problem with the minimization of $(3)$ is that for IIR filters, it results in a non-linear optimization problem, which is much harder to solve directly, and which might also have locally optimal solutions that are far from the global optimum. The cost function $\hat{C}$ in your question is linear in the filter coefficients. The point of the iteration is now to solve a sequence of linear minimization problems (which is simple, just solve a system of linear equations) in order to compute the solution of the originally non-linear optimization problem (the minimization of $(3)$). Note that if convergence is achieved, then $A_{i-1}(e^{j\omega})=A_{i}(e^{j\omega})$, so the cost function $\hat{C}_i$ is identical to the cost function of the original non-linear problem. Yet, only linear minimization problems are solved in each iteration.

I'm not sure I completely understand your last question, but the weighting function is there to change to problem from minimizing the average squared difference between frequency responses to the average squared differences between the logarithms of the magnitude responses. The necessary weighting function is unknown, but if the procedure converges - note that there is no guarantee that it does in all cases - the final weight function is such that the mean squared logarithmic error is minimized. Note that this latter problem is highly non-linear, whereas the proposed procedure only solves linear subproblems.