Need help understanding LSTMs' backpropagation and carousel of error
Part 1: Derivation

It's important to note that $\delta z \neq z^{-1}$. 

So on page 11 when he says he plans to derive $\delta I^t, \delta h^{t-1}$, and $\delta w^t$ he's invoking the chain rule rather than inverse matrix multiplication.

Let's derive $\delta I^t$.


$\delta I^t = \frac{\partial E}{\partial I^t}$ by definition
$\delta I^t = \frac{\partial z^t}{\partial I^t} * \frac{\partial E}{\partial z^t} $ by the chain rule

2.1.1 Recall that $z^t = W * I^t$

2.1.2 Therefore $\frac{\partial z^t}{\partial I^t} = W^T$

2.2.1 $\frac{\partial E}{\partial z^t} = \delta z^t$ by definition
Therefore $\delta I^t = W^T * \delta z^t$


The author continues by noting that since $I^t = \begin{bmatrix}x^t\\h^{t-1}\end{bmatrix}$, that $\delta h^{t-1}$ can be retrieved from $\delta I^t$ where  $\delta h^{t-1}$ would be the dx1 terms after the first nx1 terms.

Along the same vein to derive $\delta W^t$, 


$\delta W^t = \frac{\partial E}{\partial W^t}$ by definition
$\delta W^t = \frac{\partial E}{\partial z^t} * \frac{\partial z^t}{\partial W^t} $ by the chain rule

2.1.1 Recall that $z^t = W * I^t$

2.1.2 Therefore $\frac{\partial z^t}{\partial W^t} = (I^t)^T$

2.2.1 $\frac{\partial E}{\partial z^t} = \delta z^t$ by definition
Therefore $\delta W^t = \delta z^t * (I^t)^T$


Part 2: Constant Error Carousels

The following explanation draws heavily from the 14th slide of this lecture, and from the LSTM section of this blog, which I would highly recommend referencing. Consider this portion of my answer the TL;DR version of these two links.

The Constant Error Carousel (CEC) as you might well know is the magic of the LSTM in that it prevents vanishing gradients. It's denoted as follows:

$c_{t+1} = c_t *$ forget gate + new input $*$ input gate

In the case of regular RNNs during backpropagation, the derivative of an activation function, such as a sigmoid, will be less than one. Therefore over time, the repeated multiplication of that value against the weights $f'(x) * W$ will lead to a vanishing gradient. 

In the case of an LSTM, we only multiply the cell state by a forget gate, which acts as both the weights and the activation function for the cell state. As long as that forget gate equals one, the information from the previous cell state passes through unchanged. 

So in our case, the parameters we derived $h_t$ and $h_{t-1}$ are the hidden states which as part of the input vector $I^t$, will be filtered through the forget gate, and affect the cell state accordingly. The $\delta W$ we solved for is the parameter we use to update all our weights/gates.