Why should $a,b < N$ for Montgomery Reduction?
Looks like misunderstanding to me: you are asking about Montgomery reduction but your example is about Montgomery multiplication.

Montgomery reduction inputs a number in range $[0..NR-1]$ and outputs a number in range $[0..N-1]$, where $R$ - auxiliary Montgomery modulus.

Montgomery multiplication inputs two numbers $A$ and $B$ in range $[0..N-1]$ and outputs a number in range $[0..N-1]$

Internally, Montgomery multiplication is standard integer multiplication followed by Montgomery reduction; so, formally it is possible to apply Montgomery multiplication to any numbers $A \ge 0$ and $B \ge 0$ such that $AB \le NR-1$. Since $R \gt N$, this is weaker requirement than $0 \le A \le N-1$ and $0 \le B \le N-1$, but I doubt this possibility makes practical sense.