Self-organizing networks such as Neural Gas, Growing Neural Gas and many others have been adopted in actual applications for both dimensionality reduction and manifold learning. Typically, the goal in these applications is obtaining a good estimate of the topology of a completely unknown subspace that can be explored only through an unordered sample of input data points. In the approach presented here, the dimension of the input manifold is assumed to be known in advance. This prior assumption can be harnessed in the design of a new, growing self-organizing network that can adapt itself in a way that, under specific conditions, will guarantee the effective and stable recovery of the exact topological structure of the input manifold.
@cite_3: We investigate the convergence properties of the self-organizing feature map algorithm for a simple, but very instructive case: the formation of a topographic representation of the unit interval [0,1] by a linear chain of neurons. We extend the proofs of convergence of Kohonen and of Cottrell and Fort to hold in any case where the neighborhood function, which is used to scale the change in the weight values at each neuron, is a monotonically decreasing function of distance from the winner neuron. We prove that the learning dynamics cannot be described by a gradient descent on a single energy function, but may be described using a set of potential functions, one for each neuron, which are independently minimized following a stochastic gradient descent. We derive the correct potential functions for the oneand multi-dimensional case, and show that the energy functions given by Tolat (1990) are an approximation which is no longer valid in the case of highly disordered maps or steep neighborhood functions.
@cite_8: A neural network algorithm based on a soft-max adaptation rule is presented. This algorithm exhibits good performance in reaching the optimum minimization of a cost function for vector quantization data compression. The soft-max rule employed is an extension of the standard K-means clustering procedure and takes into account a neighborhood ranking of the reference (weight) vectors. It is shown that the dynamics of the reference (weight) vectors during the input-driven adaptation procedure are determined by the gradient of an energy function whose shape can be modulated through a neighborhood determining parameter and resemble the dynamics of Brownian particles moving in a potential determined by the data point density. The network is used to represent the attractor of the Mackey-Glass equation and to predict the Mackey-Glass time series, with additional local linear mappings for generating output values. The results obtained for the time-series prediction compare favorably with the results achieved by backpropagation and radial basis function networks. >
As proven in @cite_8 , the NG algorithm obeys a stochastic gradient descent on a function based on the average of the geometric quantization error. As known, this is not true of a SOM @cite_3 , whose learning dynamics does not minimize an objective function of any sort. This property of NG relies on how the units in the network are adapted: at each input, the units in the network are first sorted according to their distance and then adapted by an amount that depends on the ranking.