Why is NO2 formed when N valency is 3
The rule of swapping the valencies of the atoms to get their ratios only works for a very specific set of compounds and is more of a convenient trick than a rule based on any sort of hard theory.

In particular, it works for binary 'ionic' compounds such as $\ce{NaCl}$, $\ce{CuCl2}$, $\ce{Al2O3}$ etc. because the valencies of the atoms correspond to the charges on the ions and the net charge has to be zero.

However, it obviously doesn't work for a huge range of other compounds where there is more complicated bonding. For example there are a whole range of covalent nitrogen oxides of varying degrees of stability which do not fit the rule ($\ce{N2O}$, $\ce{NO}$, $\ce{NO2}$, $\ce{N2O3}$, $\ce{N2O4}$ etc).

Ultimately the only way to determine the formula of any arbitrary substance is through experiment, typically with a mass spectrometer.

If you are interested in finding out more about the different types of substances that are possible I suggest that you start reading about different types of bonding, in particular ionic versus covalent, to begin to understand the different types of compounds that exist.