When does automaton stay unchanged after string homomorphism?
Let $\mathcal{A} = (Q, A, \cdot, q_0, F)$ be the minimal DFA of your language $L$. If I understand correctly, you would like to have a permutation $\sigma$ on $A$ such that
$$
q \cdot \varphi(a) = q \cdot \sigma(a)
$$
for every state $q$ and letter $a$. This will happen if and only if, there exists $\sigma \in \mathfrak{S}(A)$ such that, for each letter $a$, 
$$
\varphi(a) \in \bigcap_{q \in Q} L(q, q \cdot \sigma(a))
$$
where, given two states $p$ and $q$, $L(p, q) = \{u \in A^* \mid p \cdot u = q\}$.

For instance, in your example, you get the following conditions when $\sigma$ is the identity
\begin{align}
 \varphi(a) \in L(0,0) \cap L(1,1) &amp;= \{ u \in A^* \mid |u|_b \text{ is even }\} \\
 \varphi(b) \in L(0,1) \cap L(1,0) &amp;= \{ u \in A^* \mid |u|_b \text{ is odd }\} \\
\end{align}
and the following conditions when $\sigma$ is the transposition $(1\ 0)$
\begin{align}
 \varphi(a) \in L(0,1) \cap L(1,0) &amp;= \{ u \in A^* \mid |u|_b \text{ is odd }\} \\
 \varphi(b) \in L(0,0) \cap L(1,1) &amp;= \{ u \in A^* \mid |u|_b \text{ is even }\} \\
\end{align}
which, altogether, gives your condition.