Abstract
Abstract. In this article, we present a lattice di erential equation model for a class of neural networks. We de ne a subset of the equilibrium solutions we call mosaic equilibrium solutions. Existence and stability theorems are proved for mosaic equilibrium solutions. Regions of stability are de ned and spatial entropy calculations, as a measure of the complexity of the system, are presented that give insights in to the e ects of spatial coupling.

1. Introduction. Neural networks are computational models characterized by patterns of weighted interconnections between neurons or cells. The method of determining the weights is called a training algorithm which resets the weights in accordance with some activation function. The result is a system which trains itself to recognize patterns or emulate functions. Traditional nets such as the Hop eld Net and the standard Backpropagation Neural Network have been intriguing to many disciplines. Although training can be slow, the resulting network can be very useful in real-time situations. Rigorous analysis of some neural network models has been done to verify the convergence of such systems 14] and to nd appropriate learning algorithms to meet the needs of varied applications 11]. Results verifying the existence of stable equilibria have been generally accepted. However, problems persist in many areas in terms of convergence rate and misclassi cation. Some researchers have approached the problem through statistical methods and probability theory 2] or methods in electrical engineering theory 13]. We take another look at the underlying mathematical model of various networks and evaluate the computational model in terms of spatial entropy, which may be considered a measure of the system's complexity. Typically, results and comparisons with reaction-di usion models leave the impression that chaotic behavior is possible, if not predictable. Hop eld nets have been modeled with reaction-di usion equations and backpropagation networks have been found to be sensitive to initial conditions and to behave in accordance with the existence of strange attractors. Since lattice di erential equations have been studied and theorems for their existence and uniqueness proved, the use of techniques associated with these systems seems well suited to the task of developing a general neural network model. This enables our analysis to include spatial entropy, as a measure of the system complexity, which can be calculated. This allows the e ect of spatial coupling to be explored.

In this article we consider a general model of a class of neural networks. The model we consider is with 2 L representing the spatial variable. We take the function f in (1.1) to be f(x( )) = 8 < :

x( ) jx( )j < 1 x( ) 2 1; 1) ?

x( ) 2 (?1; ?1] bias term for each neuron is given by . This model contains arbitrary couplings through both x and f(x) dependent on space.

We prove the existence of equilibrium solutions called mosaic solutions. Mosaic solutions are equilibrium solutions that belong to a speci c, enumerated set. These solutions correspond to x 2 f(?1; ?1]; 0; 1; 1)g which give f(x ) 2 f? ; 0; g. Through the establishment of stability criteria we will be able to de ne regions of stability. We will then de ne these regions for a general one-dimensional system, where we take ( ) = f ? 1; + 1g . We will calculate spatial entropy to get a measure of the complexity of the parameter space of (1.1) and (1.2). Our contribution is to provide conditions for existence and stability of mosaic equilibrium solutions of (1.1) and (1.2) and calculate the spatial entropy of the general one-dimensional system. Using the existence and stability criteria we calculate the spatial entropy of the resulting general system. This model, which represents a general class of neural networks, will allow us to see the e ect of spatial coupling coe cients in such systems. This will give us some insight into the e ects of parameter choices in general and allow us to examine speci c cases. We also look at the symmetric one-dimensional case to determine the e ect of symmetric coupling coe cients by using the existence and stability criteria to calculate the spatial entropy of the symmetric one-dimensional system. Since neural networks are wellknown as gradient systems and their convergence to stable points has been well established 14], we hope to add an understanding of the role played by the parameters in determining system complexity. We want to examine the e ect of the coupling parameters and .

The model, (1.1) and (1.2), may be considered a lattice di erential equation. We use the techniques of 3] to analyze the model and determine the spatial entropy of the system, which can be thought of as the measure of the size and complexity of a nonempty set of mosaic solutions. The work of Chow, Mallet-Paret and Van Vleck gives us a basis for this analysis. In 3], they describe the time evolution of such a system as a lattice di erential equation. Given L is the set of discrete points comprising the lattice, we take 2 L such that u is an in nite vector representing the state of the dynamical system. This system has the form _ u = g(u), where the _ u is taken to be the derivative with respect to time. for a one-dimensional lattice and a two-dimensional lattice with denoting the discrete Laplace operator. The existence and stability of mosaic solutions is given and an approximation of the basins of attractions made through a calculation of the spatial entropy. How and when to apply the networks and the choice of weights, bias and learning rate is a continual topic of discussion.

We seek to add an analysis of the spatial entropy of a system representing a class of neural networks in general terms to this existing work. We provide a basis for exploration of the stability in non-symmetric cases as well as the symmetric case. By making a general model, we hope to lay a foundation for studying the spatial entropy of a large class of neural networks. In this context we hope to explore the e ect of the spatial coupling on the dynamics of the system. This paper includes existence and stability theorems and their proofs, a general one-dimensional analysis of spatial entropy, application to speci c neural networks and conclusions as to our results. In Section 2 we de ne mosaic solutions, and present and prove theorems that establish the existence and stability of mosaic solutions. In Section 3 we de ne spatial entropy and give a basis for the calculations in the one-dimensional case. In Section 4 we calculate general results for a system de ned by (1.1) and (1.2). Finally, in Section 8 we draw conclusions from our exploration and give possible future directions.

2. Existence and Stability Theorems. In the model we consider, (1.1) and (1.2), no initial assumptions of symmetry or relative magnitude of the or is made to allow the examination of many neural networks. In this chapter we de ne mosaic solutions and prove their existence and establish stability criteria.

We de ne mosaic solutions to be equilibrium solutions of (1.1) and (1.2) where for every in L we have x( ) 2 f(?1; ?1]; 0; 1; 1)g. These will yield f(x( )) 2 f? ; 0; g.

To help simplify many equations, we introduce some additional notation. First, let

This term contains all the function values and the constant term. This generally is the activation of the neurons and bias term. Also, let

This term has all the references to x( ) alone. This is the initial state of the neurons. Now, we state a theorem to establish the existence of a mosaic solution. Proof:

Assume that x( ) is a mosaic equilibrium solution. Suppose that 0 < < 1 and 0 < < 1. Then we will take a v 2 N(x; ; ). That is, v 2 (x ? ; x + ) for jx j 1 and v 2 (x ? ; x + ) when

Then for any v 2 N(x; ; ) the following is de ned

where f is de ned by formula (1.2). Now for a -neighborhood of x( 0 ), we want to bound f(x( 0 ) ) as a value at the right endpoint or a value at the left endpoint depending on whether x is in the set 1; 1) or in the set (? With these inequalities established, we consider three cases: x( ) = 0, x( ) 1, and x( ) ?1. We will consider each case by examining the value of the derivative of v at each endpoint. We expect the value at the right endpoint to give us a negative derivative value and the value at the left endpoint to give us a positive derivative value. This will guarantee that the values in the or neighborhoods will remain within the prescribed neighborhood indicating stability. This will ensure that we have the stability conditions met when jx 0 j > 1 , since we can take su ciently small for (2.14) and su ciently small for (2.15) to meet (2.2), (2.3, (2.4) and (2.5).

3. Spatial Entropy. Spatial entropy is a measure of the spatial disorder of a set. Spatial ordering is the ability of the set to exhibit pattern formation as opposed to exhibiting a disordered structure. The spatial entropy, h(U) 0, measures the number of di erent patterns discernible among the elements of U in nite subsets of the lattice on which the set is de ned. Larger values of h(U) imply greater spatial complexity in U. In 3] , the spatial entropy is calculated by a technique based on a Markov shift, using a transfer matrix method to compute the entropy of the Markov shift.

In general, the spatial entropy of h(U) can be derived from a construction of successive stages of a To calculate the entropy, we consider the transfer of one stable mosaic equilibrium to another while going from one lattice point to its adjacent lattice points. If this transformation again results in a stable mosaic equilibrium, the entropy of the region is higher than if there were not another stable point. We use a method of Markov Shifts, which depends on building boolean matrices, that is matrices whose elements are 0 or 1. These transfer matrices indicate the possibility of a transfer from one equilibrium mosaic solution to another. For eligible points a sequence of overlap values is obtained and serves as the row and column de nitions. The element value 1 marks a valid transfer from one equilibrium con guration to another. The zero indicates no such possibility. In the one-dimensional case the process is very well-studied and an explicit calculation of the entropy is well de ned. Using the transfer matrix We de ne U = S ( ; ) to be the set of stable mosaic solutions of (4.1) and (1.2) for the onedimensional case. We are interested in how the parameters ( ; ) a ect the spatial entropy. We are looking for dependencies that give entropy values of zero as opposed to those where the entropy value is positive. As de ned by 3], our system exhibits spatial chaos when h(S ( ; )) > 0 and exhibits pattern formation when h(S ( ; )) = 0.

The following tables relate the existence conditions and the stability criteria to the speci c triples. They are separated into three categories by the similarity of the values contained within each triple. For each possible triple in each stability region, the transfer condition (5.1) is speci c to a triple pair that is compatible to the transfer. There are eight regions of stability de ned for speci c groups of triples. By compiling the stability criteria from Tables 4.1 For transfer to occur, the last term of the rst triple, T 1 , must be equal to the rst term in the second triple, T 2 . These eight regions de ned in Tables 4.1 We determine the composite stability regions by correlating the region from the left with a column designated above. The dots represent the compatibility of the third element of the triple T 1 2 Un 1 (along the left) and the rst element of the triple T 2 2 Um 2 (from above). If transfer is possible we nd the composite region Un 1 ! Um 2 which will be called Rp.

These regions can be explicitly described by the formula:

(1) + (2) < A + B + C + D (5.2) where A = ?jL (1) ( )j(jL (1) ( ? 1)jj (1) ?1 j ? jL (1) ( + 1)jj (1) +1 j) B = ?jL 2 ( )j(jL (2) ( ? 1)jj (2) ?1 j + jL (2) 

For example, a triple from U1 1 can transfer to a triple from U1 2 giving Region 1. That is, the triple (1; 1; 1) ! (1; 1; 1) is a valid transfer. Using this in the formula above with L de ned in 2, The stability region is 1 + 2 < ?j (1) ?1 j ? j (1) +1 j ? j (2) ?1 j ? j (2) +1 j There are 32 regions described above, then, each having speci c triples that have existence and stability properties. By taking the eligible triples, we can extract the doubles that represent the overlap between those triples where the transfer conditions hold. These doubles can be ordered to form transfer matrices. The ordered list of doubles serves as the row names and column names to construct the transfer matrix. An association of the row double and a compatible column double yields a 1 in the associated entry of the matrix. That is, the pattern must be possible and the result of the type represented by the stability region from which the original triples were extracted. If the row double is not compatible with the column double or the resulting triple is not admissible to the region, the associated matrix entry is 0.

The eigenvalue of such a matrix is then calculated. We identify the possible triples to make a transfer possible in R1. The largest eigenvalue of this matrix is 2. Thus the entropy for this region is ln(2).

Appendix A contains the general transfer matrix for all 32 regions. The entropies for the regions are summarized in the Table 5 .2. When the eigenvalue of the matrix is 0 we cannot calculate ln(0), hence we say the entropy is unde ned. Frequently in this case the transfer matrix might contain all zeros, since the pattern formed by the overlap does not exist in the region under consideration. There are three regions in this general case where the entropy indicates the existence of spatial chaos. There are also three regions of pattern formation.

Entropy value Regions 0 R28, R31, R32

ln (2) R1, R2, R13 unde ned all others Table 5.2 Entropy and Regions 6. The Symmetric One Dimensional Case. With our restrictions for the symmetric one-dimensional case the existence criteria can be represented as lines in the ?1 plane. The existence lines can then be examined as they cross the stability regions. It is clear that for existence and stability to coexist, the intercepts of the existence lines must be less than zero. That is, =x < 0. We'll calculate the entropy for the stability regions formed when ? < 0. Furthermore, we'll consider the case where =x < ? . Given these further constraints, the admissible triples for each region Sn can be determined. Regions L1 L2 L3 L4 L5 L6 L7 S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 To calculate the entropy we consider the transfer of one stable equilibrium to another while going from one lattice point to its adjacent lattice points. If the transformation again yields an existence point ( a possible triple in the region under consideration), then there is another point of stability and the entropy of the region is higher than if there were not another possible stable point. Using a Markov Shift we must rst construct a transfer matrix. This is a matrix of boolean values representing the possibility of a transfer of stable points. The natural logarithm of the largest eigenvalue of the transfer matrix is the entropy of the region.

Complete calculations are in Appendix B. We summarize the results in Table 6 .2. If the eigenvalue of the transfer matrix is 0, the resulting entropy value is unde ned.

Four regions show a high entropy value indicating the presence of spatial chaos. Eight show regions of pattern formation.

7. Comparison of the General and Symmetric One-dimensional Cases. When we compare the entropies of the regions in the symmetric case with those of the general case some signi cant correlations appear. The regions S2 and S9 in the symmetric case have the same entropy level as R1, Entropy value Regions 0 S3, S4, S7, S8, S11, S12, S15, S16

ln (2) S2, S9 < ?2j ?1 j and > ? (7. 2)

The general form of the boundaries are very similar. The restrictions in the symmetric case reduce to a single region by virtue of very strict assumptions. For this initial comparison we have restricted the intercept of existence lines, < ? . In the symmetric case the inequality may be seen as representing the overlap of the general cases. Since in the general case the same number of admissible triples is originally present, the entropy in the symmetric case is not any higher than in the general case.

The symmetric case re ects a dependence on the value of =x in relation to the value of ? . Region S9 is a region on the edge of chaos. Depending on the value of ? alone, it has relatively high entropy or is a region of no pattern formation.

However, if we look at those regions in the symmetric case of high entropy which were not evident in the general case, it is possible to form a reasonable composite region that could be present in a symmetric or otherwise restricted case. Consider S1 in the symmetric case. The stability region is given by < ? ? 2j ?1 j. We nd the resulting entropy to be ln(2:89), very close to the maximum expected for a solution set of three elements. In building a composite region from the general regions, we look for those elements that would naturally lead to a similar stability form. R1, R10, R11, R12 all have references to the j ?1 j-type terms. R19, R20, R23, R24, R27, R28, R29, R31, and R32 all have the sum of the two ? -type terms. The resulting transfer matrix for this constructed composite region has entropy ln(3). This indicates that it is possible for the symmetric case to e ectively merge general regions into composite regions with higher entropy. 8 . Conclusion. We can conclude from the previous analysis, that there is a basis for describing chaotic behavior. Regions in a general one-dimensional parameter space have been identi ed that have a spatial entropy value in excess of zero. These regions of spatial chaos tend to leave the neural network model sensitive to initial conditions. Coupling coe cients and connectivity do play a part in the behavior of the time-evolutionary system. In the general one-dimensional case we nd 32 regions de ned, 3 regions have entropy greater than zero, the value calculated as ln (2) . For mosaic solution sets of 3 values the largest entropy is ln(3) making those values of ln(2) an indicator of spatial chaos.

Symmetric coe cients increase the chances of chaotic behavior by raising the entropy calculation in predictable regions. We found 17 distinct regions where the largest entropy value is ln(2:89) in one region. Three other regions have entropy greater than 0. This indicates higher spatial chaos in the symmetric case than in the general case. The number of distinct regions is smaller due to the symmetry of the coe cients. This analysis presents stability conditions and the regions formed by these conditions. The spatial entropy analysis indicates the signi cant possibility of chaotic behavior. More remains to be done to make this analysis applicable to creating neural networks and using this information to streamline existing models. Currently, it is understood that random initial weights avoid problems of convergence to local minima as opposed to global minima. But more research into the e ects of the weight-related coe cient needs to be done. The role of the bias term has not been explored, as it could manage the term and hence the term. We see a further dependence in the symmetric case where region S9 is a region of high entropy or a region of non-stability depending on the value of and the value of . A similar analysis could be done in terms of and as well. More experimental research into this dependence is certainly indicated to enhance the intuition regarding these results.

We also leave for future research the explicit analysis of those systems of higher dimension. Many neural networks are of much higher dimension than the one-dimensional case. In general, those methods necessary to analyze regions of higher dimension are still in development. Explicit region de nition and entropy calculation for those models of higher dimension will give better understanding of the roles of the coupling coe cients and the bias terms. The technique will be best tested when applied directly to a Hop eld net, a CNN, a Backpropagation, or perhaps a Kohonen formula.

Appendix A .

In this appendix, we present a general matrix for the 32 regions de ned in the general one-dimensional case. The entropies are summarized in a table and the regions are de ned in Chapter 4.

The eligible doubles in order of rows and columns of the matrices are: In this appendix, we present general transfer matrix in the symmetric one-dimensional case. A summary of the regional de nitions and resulting entropies is presented in Chapter 4.

The sequence of admissable doubles used to construct the transfer matrix is as follows: The values are 1 in the elements given for the symmetric matrices S1 -S17: S1 => a 11 ; a 12 ; a 13 

{'paper_id': '207121790', 'title': 'Mosaic Solutions and Spatial Entropy for a Class of Neural Networks Models', 's2_url': 'https://api.semanticscholar.org/CorpusID:207121790', 'source': 'S2ORC'}