Why is there no "traditional"-mathy way to describe the general algorithm and give a more math-friendly definition of algorithm?
If you're looking for algebraic structure, then you should look at the field of denotational semantics. This is exactly what you describe: using algebra, and often Category Theory, to model computation mathematically.
Some examples:

Domain theory provides a mathematical model of the untyped lambda calculus, which is powerful enough to capture all computable functions. The trick is that, to model computation, we need to represent functions, but for any set $S$, the function space $S \to S$ is always larger than $S$. So there's no simple set that captures all computations involving computations. Instead, Dana Scott introduced domains, which impose additional structure on sets to avoid countability problems.

SKI Combinators provide a &quot;point-free&quot; description of computation. Every computation can be expressed by composing and applying three base combinators $S$, $K$, and $I$. The equations for the combinators are:

$ S\ x\ y\ z = (x\ z)(y\ z)$
$ (K\ x) y = x$
$ I x = x$

So these equations, in some sense, give an algebraic account of all of computation.

Cartesian closed categories provide models for typed programming languages. They don't capture all of computation, but they do provide an abstract model of it.

Topos theory provides a generalization of logic using category theory. And logic corresponds to computation via the Curry-Howard correspondence


As for recursion, it turns out that you can eliminate it by instead using fixed-point combinators like the Y combinator. So recursion can always be eliminated from (untyped) languages, even without using iteration.