Mathematical definition of a symmetry operator
That's not entirely correct. You're making at least an extra assumption here. You're assuming that $\hat{R}$ is a function from $D\mapsto D$ where $D$ is the domain of $f$, but there's no reason why $f$ must operate as a function on $D\mapsto D$. It might just as well be $f: D\mapsto \mathbb{R}$.
I think what you're really asking for is that a symmetry element $\hat{R}$ such that $\hat{R}: D\mapsto D$, and $\forall x\in D$, $f(x) = f(\hat{R}x) = (f\circ \hat{R})x$.
Then for example, for $f(x) = (\cos x, \cos x)$, the transformation $\hat{R}$ mapping $x \rightarrow -x$ would would a valid symmetry element.