In RNNs why do networks always use the last output vs the last input?
That would work if the Markov Assumption was in place here (i.e. the last state was enough to determine the new state). 

However, that is exactly what RNNs try to break. RNNs try to encode the whole history of inputs and predictions to determine the next output.

So in the formula, the $h_{t-1}$ is the last output, but it is a hidden state which encodes all the previous inputs, outputs, and calculations. So when you have new information coming in (i.e. $x_t$), your model can make use of all the history and the new input to determine the new input.

The same applies to LSTM, you are working with a sequence that the new output depends not only on the last input but on more things from the past, and this information is encoded in $h_{t-1}$.