Gibbs free energy of formation vs. Gibbs free energy of reaction?
What is the origin of the second formula (2)?
  


Equation (2) evidently refers to the free energy of formation (at standard pressure) of a solid solution of A and B, in which case neither element is in its standard state as a product (so that $a_i\neq 1$). You can rewrite expression (2) as follows:

$$\Delta G_\mathrm{f}^\circ = nRT\ln{\left(a_\ce{A}^{x_\ce{A}}a_\ce{B}^{x_\ce{B}}\right)} = RT\ln{\left(a_\ce{A}^{x_\ce{A}}a_\ce{B}^{x_\ce{B}}\right)^n}= RT\ln{\left(a_\ce{A}^{nx_\ce{A}}a_\ce{B}^{nx_\ce{B}}\right)}= RT\ln{\left(a_\ce{A}^{n_\ce{A}}a_\ce{B}^{n_\ce{B}}\right)}$$

$$ = n_\ce{A}RT\ln{a_\ce{A}+n_\ce{B}RT\ln{a_\ce{B}}}$$

where $n$ is the total number of moles ($n=n_A+n_B$).

Since the activity of the elements in pure solutions is 1 we can finally write

$$\Delta G_\mathrm{f}^\circ = n_\ce{A}\Delta G_{mA}+n_\ce{B}\Delta G_{mB}$$
where $\Delta G_{mi} = G_{mi}-G_{mi}^\circ= n_\ce{i}RT\ln{\left(\frac{a_\ce{i}}{a_\ce{i}^\circ}\right)}$.

However the most useful way to express this is as follows:

$$\Delta G_\mathrm{f}^\circ = (n_\ce{A}G_{mA}+n_\ce{A}G_{mB})-(n_\ce{B}G_{mA}^\circ+n_\ce{B}G_{mB}^\circ)$$

In words, the free energy change corresponds to mixing of $n_\ce{A}$ moles of pure A and $n_\ce{B}$ moles of pure B (the pure components having activity $a_i^\circ=1$) to form a mixture where the components have activities $a_i$.  


  
  How does it come, that in one case the activity of the whole product AB is important and in another the single activities of the components of the product?
  


Equation (1) refers to a molar free energy of formation of $\ce{AB}$ from reagents $A$ and $B$, all under the same constant (P,T, composition) conditions, whereas (2) refers to the free energy of formation of a solid solution of $n_A$ moles of A and $n_B$ moles of B from pure components. Equation (1) refers to combination of A and B at a 1:1 mole ratio, or equivalently reaction to form 1 mole of $\ce{AB}$ from $n_A=n_B=\pu{1 mol}$. Reaction (2) refers to mixture of A and B at any arbitrary ratio or total number of moles. Therefore equation (2) is in a way more general. Also, equation (1) refers to a differential process (transformation to form 1 mole of product under contant conditions) whereas (2) refers to an integral (mixing) process.  




  For the given reaction:
  $$\Delta G = \Delta G^⦵ + RT\ln{\frac{a_\ce{AB}}{a_\ce{A}\cdot a_\ce{B}}} $$
  Since all components are pure solid substances, all activities equal 1 and therefore, $\Delta G = \Delta G^⦵$.


A comment on this: the fallacy here is to assume that $a_i=1$ at equilibrium. Only for the pure reagents and products in their standard states it is strictly required by definition that $a_i=1$.

Note also that since $\Delta G^\circ = G_{mAB}^\circ-G_{mA}^\circ-G_{mB}^\circ$ and  $G_{mi} = G_{mi}^\circ+RT\ln{\left(\frac{a_\ce{i}}{a_\ce{i}^\circ}\right)}$ we can write

$$\Delta G = G_{mAB}-G_{mA}-G_{mB}$$

In words (and repeating myself), $\Delta G$ in this case is for the process of converting 1 total mole of $A$ and 1 mole of $B$ into 1 mole of $AB$, all at constant T, P and composition (or, equivalently, in a sufficiently large mixture, such that a 1 mole change in the amount of the substances does not change the properties).

Schematically, this is how I interpret the two processes, with equation 2 for mixing in the top, and the reaction of A and B to form AB in the bottom: