Parallel or efficient computation of value with momentum
I have noted that the above expression can be rewritten as:

$$
A'_{i,j} = \alpha^j A'_{i,0} + \sum_{1 \leq k \leq N} (1 - \alpha) \alpha^{j-k}A_{i,k}
$$

Which makes me think I can define a $M \times N$ matrix $P$ as:

$$
P_{i,j} = \alpha^j A'_{i,0}
$$

And a $N \times N$ matrix $R$ as:

$$
R_{i,j} =
\begin{cases}
\alpha^{j-i} &amp; \text{if}~i \leq j \\
0 &amp; \text{otherwise}
\end{cases}
$$

And so I think that I would have:

$$
A' = P + (1 - \alpha) A R
$$

Which would be a more parallelizable computation. The only challenges would be to efficiently construct $P$ and $R$ (which is probably not that hard) and, more importantly, the memory usage if $N$ is big (which unfortunately in my case it kinda is).