where η is the learning rate, D is the desired output, and Y is the actual output.
This is called the Perceptron Learning Rule, and goes back to the early 1960's.
This restricts the possible mappings available.
is a development from the simple Delta rule in which extra hidden layers (layers additional to the input and output layers, not connected externally) are added. The network topology is constrained to be feedforward: i.e. loop-free - generally connections are allowed from the input layer to the first (and possibly only) hidden layer; from the first hidden layer to the second,..., and from the last hidden layer to the output layer.
The hidden layer learns to recode (or to provide a representation for) the inputs. More than one hidden layer can be used.
The architecture is more powerful than single-layer networks: it can be shown that any mapping can be learned, given two hidden layers (of units).
The graph shows the output for k=0.5, 1, and 10, as the activation varies from -10 to 10.
The weight change rule is a development of the perceptron learning rule. Weights are changed by an amount proportional to the error at that unit times the output of the unit feeding into the weight.
the outputs are calculated and the error at the output units calculated.
The output unit error is used to alter weights on the output units. Then the error at the hidden nodes is calculated (by back-propagating the error at the output units through the weights), and the weights on the hidden nodes altered using these values.
For each data pair to be learned a forward pass and backwards pass is performed. This is repeated over and over again until the error is at a low enough level (or we give up).
Radial basis function networks are also feedforward, but have only one hidden layer.
Like BP, RBF nets can learn arbitrary mappings: the primary difference is in the hidden layer.
RBF hidden layer units have a receptive field which has a centre: that is, a particular input value at which they have a maximal output.Their output tails off as the input moves away from this point.
Gaussians with three different standard deviations.
training up the output layer.
Generally, the centres and SDs are decided on first by examining the vectors in the training data. The output layer weights are then trained using the Delta rule. BP is the most widely applied neural network technique. RBFs are gaining in popularity.
trained on classification data (each output represents one class), and then used directly as classifiers of new data.
trained on (x,f(x)) points of an unknown function f, and then used to interpolate.
Networks of this form (finite-impulse response) have been used in many applications.
There are also networks whose architectures are specialised for processing time-series.
Simple Perceptrons, BP, and RBF networks need a teacher to tell the network what the desired output should be. These are supervised networks.
In an unsupervised net, the network adapts purely in response to its inputs. Such networks can learn to pick out structure in their input.
Although learning in these nets can be slow, running the trained net is very fast - even on a computer simulation of a neural net.
- takes a high-dimensional input, and clusters it, but retaining some topological ordering of the output.
After training, an input will cause some the output units in some area to become active.
Such clustering (and dimensionality reduction) is very useful as a preprocessing stage, whether for further neural network data processing, or for more traditional techniques.