How to choose a fixed adaptation step for decision feedback equalizer
I'm leaving the answer here, if somebody ever stumbles upon the same question.
About LMS: Both DFE and ARC employ Least Mean Squares (LMS) adaptive algorithms: DFE is an adaptive filter and ARC can be considered as an additional tap of that filter. Hence the adaptation step size should be selected accordingly.
The general update equation for the LMS algorithm is the following:
$$
\mathbf{w}(n+1) = \mathbf{w}(n) + \mu\cdot e(n) \cdot \mathbf{x}(n),
$$
where $\mathbf{w}(n)$, $e(n)$ and $\mathbf{x}(n)$ are respectively filter weights, an error and tap values at $n^{th}$ sample and $\mu$ can be a constant scalar.
With $p$ being the filter order, the bounds for $\mu$ can be defined as
$$
0 &lt; \mu &lt; \frac{2}{(p+1)E\{|x(n)|^2\}},
$$
where $E\{|x(n)|^2\}$ represents the power in $x(n)$ and can be estimated using an average:
$$
\hat{E}\{|x(n)|^2\} = \frac{1}{N}\sum_{k=0}^{N-1} |x(n-k)|^2.
$$
Then $\mu$ value can be chosen as a fraction of $\mu_{max}$, e.g. $0.125 \cdot \mu_{max}$.
However, in the sign-error LMS, which is an LMS algorithm with reduced complexity with the following update equation:
$$
\mathbf{w}(n+1) = \mathbf{w}(n) + \mu\cdot sign(e(n)) \cdot \mathbf{x}(n),
$$
replacing an error value with its sign changes the magnitude of the correction. If $e(n)&gt;1$, then the magnitude of the correction is smaller compared to the standard LMS. Hence the $\mu$ value can be chosen to be larger.
To the questions:

For example in a question, the input signal is a sequence of evenly (presumably) distributed PAM4 symbols, i.e. &quot;-3&quot;, &quot;-1&quot;, &quot;1&quot; and &quot;3&quot;, which gives a power equal 5. This gives $\mu_{max} = \frac{2}{(p+1)\cdot 5}$.

DFE and ARC employ different LMS algorithms: sign-error LMS and standard LMS respectively. This calls for separate $\mu$ values for DFE and ARC. Hence the recommended adjustment would be to use sign-error LMS for both functions. Not only does this simplifies the $\mu$ value selection, it also simplifies the further implementation.


Considering the need to simplify the hardware implementation of the algorithms, $\mu$ can be chosen to be a power of two. For this particular example, $\mu$ equal to $\mu_{max}$ rounded to one significant binary digit happened to be a good choice.
References: A good reference on the LMS algorithms and step size selection is Chapter 9 &quot;Adaptive Filtering&quot; in Statistical Digital Signal Processing and Modeling by M.H. Hayes. Another good option is a Least_mean_squares_filter entry on Wikipedia.