How is silver chloride more soluble in brine than water?
I mean won't the common ion effect operate?


The common ion effect always applies for pure substances. With the added sodium chloride, you have changed the effective solvent and changed the ions.


  Or may be this is due to some kind of complex formation as NaCl solution is highly concentrated?


Exactly, before your ionic equation was:
$$\ce{Ag+ (aq) + Cl- (aq) &lt;=&gt; AgCl (s)}$$
but with brine, the ionic equation is:
$$\ce{AgCl (s) + Cl- (aq) &lt;=&gt; [AgCl2]- (aq)}$$
Which has a different solubility constant, and where as before more chloride made the $\ce{AgCl}$ less soluble, if you note that you have switched regimes and $\ce{Cl-}$ has switched from being a reactant to a product, which means now with additional chloride silver chloride becomes more soluble.