Shouldn't every algorithm run in pseudo-polynomial time?
The running time is always considered with respect to the length of the input.

If the input is a natural number $N\in\mathbb{N}$, then the length of the input is $n=\log N$. In that case, running time of $\Theta(N)$ is actually exponential in the length of the input, $n$.

Suppose now that we're dealing with a problem where the input is a collection of $n$ elements. For simplicity, assume each member in the collection can be represented by a constant number of bits, independent of the input. Lets denote this constant by $c$, so the number of bits required to represent this collection is at most $cn$. In that case, if upon receiving a collection with $n$ elements (which is a $cn$ length input) you preform only $O(n)$ operations, then the running time is linear.

When saying the input is a number $N$, or a list of size $n$, we already talk about the semantics of the input (how we intend to interpret the given sequence of bits). You can count the running time in terms of those "interpretations", e.g. $O(N)$ for naive primality testing, or $O(n)$ for finding a maximum in a list, but in the end you have to remember how those objects relate to the actual input's size. If we denote the length of the input by $l$, for primality testing we have $l=\log N$, so $T(l)=N(l)=2^l$, and for finding maximum $T(l)=n(l)=\frac{1}{c}l$.