External frequency divider in a PLL
Since you stated "external to the loop BW" I assume you mean specifically an output signal, not the signal fed back to the phase detector but provided to an external sink. Then yes, in this case a frequency divider would likely have phase ambiguity regardless of the division number unless special provisions are made to synchronize state (which requires an external time reference). This is because the division would start on any arbitrary edge (in typical cases the rising edge, but the salient point is any edge). Without an external synchronization of state, two identical PLL's may start the divided signal on different edges resulting in complete phase ambiguity. To eliminate this, some sort of state synchronization or reset is required.

Consider the operation of the divide by 2 as shown in the figure below. This typically operates by having the output change state on every rising edge of the input. Depending on the starting state of the local circuit, either of the two lower plots in the figure could result. Unless the circuit has a specific reference to common time there would be no way for it to remain in phase with its prior phase reference (were it to stop and start again, or considering two PLL circuits etc). This is what is meant by phase ambiguity. From this it should be clear that this would apply to any number N.