How is the Delta Rule derived in neural networks and what is the explanation for the algebra?
Your equations are wrong on several counts. First of all, the weights and the inputs are both vectors. Second, the value of actual is given by $\operatorname{sgn}(\langle \mathit{Input}, \mathit{Weights} \rangle)$, in words, the sign of the inner product of the input and the current weights. Similarly, the desired output is either $+1$ or $-1$. Therefore the error is either $+2$ (if the desired was $+1$ but the actual was $-1$) or $-2$ (in the opposite case). Third, we don't want to "solve" for the weights, since this would erase everything that we have learned so far.

The idea of the learning process is to give negative feedback. Suppose that we got $-1$ but we expected $+1$. We want to change the weights so that on the same input, $\langle \mathit{Input}, \mathit{Weights} \rangle$ will be larger. The most economical way (in some sense) is to update the weights with a new vector proportional to the input, since the input is the vector most correlated with itself (that is, the unit vector $w$ maximizing $\langle \mathit{Input},w \rangle$ is proportional to $\mathit{Input}$). Thus we increase the weights with some multiple of the input. If the error were in the other direction, we would decrease instead of increase &mdash; which is why we multiply the input by the error.