Does 'subexponential algorithm' refer to input or number of bits used to represent input?
It refers to the number of bits to represent N.  More generally, the complexity of a problem is normally taken to be a function of the length of the input (the number of bits needed to represent the input).

Trial division is exponential: its running time is $O(N^{1/2})$, i.e., $O(2^{B/2})$ where $B=\lg N$ is the length of the input in bits.  That's an exponential function (of $B$).  See Are there subexponential-time algorithms for NP-complete problems?.