How can we justify setting the affinity $\sum_i \mu_i \nu_i$ equal to zero at chemical equilibrium?
Chemical equilibrium is a stable equilibrium, i.e. the system returns to equilibrium to whatever perturbations. This also leads to the condition of equilibrium, as it should  contain this information.

The short answer to your question:

Due to the stability, the system stays/returns to equilibrium at any kind of small perturbation. It also means, $\left(\sum_i \mu_i \nu_i\right)\; d\zeta = 0$ not only in the equilibrium point, but also for any small $d\zeta$. Therefore the $\left(\sum_i \mu_i \nu_i\right)$ itself must be zero.

Somewhat longer answer:

The reaction goes from one direction to the other ($\sum_A \nu_A A \rightarrow \sum_B \nu_B B$)  spontaneously only till it reaches the equilibrium condition. In this direction  $\sum_i \mu_i dN_i \le 0$. 

We followed the reaction from a given concentration toward equilibrium. If we follow the same $\sum_A \nu_A A \rightarrow \sum_B \nu_B B$  transformation BEYOND the equilibrium point, $ \sum_i \mu_i dN_i $ must change sign at the equilibrium point, and it must be positive should be valid (since it is not a spontaneous reaction).

Therefore the condition of equilibrium is not $d\zeta = 0 $ , but sign change: $ \sum_i \mu_i dN_i =0$.