Primitive Recursive Definition : Binary numbers
The concept of primitive recursion function easily generalizes to any inductive data structure. For example, you can define the primitive recursive functions on binary strings as the smallest class of functions containing $\epsilon$ (the constant "empty string"), $S_0,S_1$ (append $0$ or $1$), projections and closed under composition and the scheme allowing to define a function $f$ by
$$
\begin{align*}
f(\epsilon,\vec x) &amp;= g(\vec x), \\
f(zi,\vec x) &amp;= h_i(z,\vec x,f(z,\vec x)), &amp; i\in\{0,1\}
\end{align*}
$$
as soon as $g$ and $h_0,h_1$ are already in the class.

In the above setting, the usual numerical functions may be encoded either by considering only unary strings (i.e., $n$ is represented by $1^n$), in which case you can mimick the usual primitive recursive functions, or, more interestingly, by representing integers in binary. In that case, the successor function is not part of the basic functions but must be defined (and this is not immediate).

As you point out, traditional recursion theory considers primitive recursion on unary strings (or tally integers) because in that setting complexity is not a main concern.  On the other hand, whenever complexity classes are defined in terms of recursive functions, primitive recursion on binary strings (or binary integers) is routinely used. See for example the seminal paper by Bellantoni and Cook "A New Recursion-Theoretic Characterization of the Polytime Functions", Computational Complexity 2(2):97-110.