How can one measure the time dependency of an RNN? Most of the discussion about RNN and LSTM alludes to the varying ability of different RNNs to capture &quot;long term dependency&quot;. However, most demonstrations use generated text to show the absence of long term dependency for vanilla RNN.
Is there any way to explicitly measure the time dependency of a given trained RNN, much like ACF and PACF of a given ARMA time series?
I am currently trying to look at the (Frobenius norm of) gradients of memories $s_k$ against input $x_l$, where $l\le k$, summed over training examples $\{x^i\}_{i=1}^N$ - $$\text{Dep}(k,l):=\sum_{i=1}^N \big\|\frac{\partial s_k}{\partial x_l}(x^i)\big\|_F$$ I would like to know if there are more refined or widely-used alternatives to this prototype.
I am working with time series so I treat the inputs $\{x_t\}$ as realization of a random process $\{X_t\}$, thus by &quot;current&quot; I mean $x_i,s_i$ for some fixed $i$, &quot;the past&quot; I mean $\{x_j\}_{j=1}^{i-1},\{s_j\}_{j=1}^{i-1}$ and &quot;time&quot; I mean the index $t$.
I guess that the &quot;long-term dependency&quot; in literature refers to the sensitivity of the current memory $s_k$ w.r.t. past inputs $\{x_j\}_{j=1}^{k-1}$, hence the prototype I formulated.
I am not aware of any standard or widely-used metric for this. I think what metric would be appropriate would depend on what you want to use it for.
The issues with RNN is &quot;forgetting&quot;.  If you feed a long sequence of inputs $x=(x_1,\dots,x_n)$ into a RNN, where $n$ is too large, the problem is that often the final decision is determined by the last few values ($\ldots,x_{n-1},x_n$) and the earliest values ($x_1,x_2,\ldots$) have been &quot;forgotten&quot; and do not affect the final decision.  This is undesirable in many settings.
Your metric would be one reasonable way to get a feeling for this.  Another reasonable way might be to to feed in an input $x=(x_1,x_2,\dots,x_n)$, then change just $x_1$ to get a new input $x'=(x'_1,x_2,\dots,x_n)$, feed in $x'$, and compare the outputs of the RNN on $x$ vs $x'$; and repeat for many training samples or test samples $x$.