Criticality & Deep Learning I: Generally Weighted Nets
Dan Oprisa Peter Toth


[

Criticality & Deep Learning I: Generally Weighted Nets

 Dan Oprisa
  Peter Toth
 

 
CriticalAI 
 http://www.critical.ai  
]



Motivated by the idea that criticality and universality of phase transitions might play a crucial role in achieving and sustaining learning and intelligent behaviour in biological and artificial networks, we analyse a theoretical and a pragmatic experimental set up for critical phenomena in deep learning. On the theoretical side, we use results from statistical physics to carry out critical point calculations in feed-forward/fully connected networks, while on the experimental side we set out to find traces of criticality in deep neural networks. This is our first step in a series of upcoming investigations to map out the relationship between criticality and learning in deep networks. 





§ INTRODUCTION


Various systems in nature display patterns, forms, attractors and recurrent behavior, which are not caused by a law per se; the ubiquity of such systems and similar statistical properties of their exhibit order has lead to the term "universality", since such phenomena show up in cosmology, the fur of animals <cit.>, chemical and physical systems <cit.>, landscapes, biological prey-predator systems and endless many others <cit.>. Furthermore, because of universality, it turns out that the most simplistic mathematical models exhibit the same statistical properties when their parameters are tuned correctly. As such it suffices to study N-particle systems with simple, "atomistic" components and interactions since they already exhibit many non-trivial emergent properties in the large N limit. Certain "order" parameters change behavior in a non-classical fashion, for specific noise levels. Using the rich and deep knowledge gained in statistical physics about those systems, we map the mathematical properties and learn about novel behaviors in deep learning set ups. Specifically we look at a collection of N units on a lattice with various pair interactions; when the units are binary spins with values (± 1), the model is known as a Curie-Weiss model. From a physical point of view, this is one of the basic, analytically solvable models, which still possesses the rich emergent properties of critical phenomena. However, given its general mathematical structure, the model has already been used to explain  population dynamics in biology <cit.>, opinion formation in society <cit.>, machine learning <cit.> and many others <cit.>. All those systems, with a rich and diverse origination, posses almost identical behavior at criticality. In the latter case of machine learning, the Curie-Weiss model encodes information about fully connected and feed-forward architectures to first order.
Similar work was done in <cit.>, where insights from Ising models and fully connected layers are drawn and applied to net architectures; in <cit.> a natural link between the energy function and an autoencoder is established.
We will address the generalisation of fully connected system and understand its properties, before moving to the deep learning network and applying there the same techniques and intuition.

The article is organised as follows: section <ref> gives a short introduction of critical systems and appropriate examples from physics; in section <ref> we map a concrete, non-linear, feed forward net to its physical counterpart and discuss other architectures as well; then we turn to investigating the practical question whether we can spot traces of criticality in current deep learning nets in <ref>. Finally we summarise our findings in <ref> and hint at future directions for the rich map between statistical systems and deep learning.



§ BRIEF SUMMARY OF CRITICAL PHENOMENA


Critical phenomena were first thoroughly explained and analysed in the field of statistical mechanics, although they were observed in various other systems, but lacking a theoretical understanding. The study of criticality belongs to statistical physics and is
an incredibly rich and wide field, hence we can only briefly summarise some few results of interest for the present article; definitely a much more comprehensive coverage can be found, see e.g.<cit.>. In a nutshell, the subject is concerned with the behavior of systems in the neighbourhood of their critical points, <cit.>. One thus looks at systems composed of (families of) many, identical particles, trying to derive properties for macroscopic parameters, such as density or polarisation from the microscopic properties and interactions of the particles; statistical mechanics can hence be understood as a bridge between macroscopic phenomenology (e.g. thermodynamics) and microscopic dynamics (e.g. molecular or quantum-mechanical interacting collections of particles). In a nutshell, criticality is achieved when macroscopic parameters show anomalous, divergent behavior at a phase transition. Depending on the system at hand, the parameters might be magnetisation, polarisation, correlation, density, etc. Specifically it is the correlation function of the "components" which then displays divergent behavior, and signals strong coordinated group behavior over a wide range of magnitudes. Usually it is the noise (temperature) which at certain values will induce the phase transition accompanied by the critical anomalous behavior. Given its relevance in physics and also its mathematical analogy to our deep learning networks, we will briefly review here the Curie-Weiss model with non-constant coupling and examine its behavior at criticality.



 §.§ Curie-Weiss model


A simplistic, fully solvable model for a magnet is the Curie-Weiss model (CW), <cit.>. It possesses many interesting features, exhibits critical behavior and correctly predicts some of the experimental findings. As its mathematics is later on used in our deep learning setup, we will briefly present main properties and solutions for the sake of self-consistency.

The Hamiltonian of the CW model is given by


    H =  -J/2N∑_ij^N s_is_j 
        - b ∑_i^N s_i


Here the s_i are a collection of interacting "particles", in our physical case, spins, that interact with each other via the coupling J; they take values (± 1) and interact pairwise with each other, at long distances; the inclusion of a factor of 1/N multiplying the quadratic spin term makes this long-range interaction tractable in the large N limit. Furthermore, there is a directed external magnetic field which couples to every spin via b. Since the coupling between spins is a constant and since every spin interacts with every other spin (except self-interactions, which is accounted by a factor of 1/2) the Hamiltonian can be rewritten to


    H =  -J/2N(∑_i^N s_i)^2
        - b ∑_i^N s_i


With β = 1/kT being the inverse temperature the partition function can be formulated


    Z    = ∑_s_i∈{± 1}e^-β H(s)
       = ∑_s_i∈{± 1}expβ[   J/2 N(∑_i^N
     s_i)^2    + b ∑_i^N s_i]


which can be fully solved, <cit.>, summing over each of the 2^N states; given an explicit partition Z, the free energy can be computed via 


    F = - kT ln Z


Once we have F various macroscopic values of interest can be inferred such as the magnetisation of the system, aka first derivative of F wrt. b. This is a so called "order parameter", which carries various other denominations, such as polarisation, density, opinion imbalance, etc. depending on the system at hand. It basically measures how arranged or homogeneous the system is under the influence of the outside field which couples to the spins via b. A full treatment and derivation of the model including all its critical behavior can be found in <cit.>, from where we get the equation of state for the magnetisation


    m = btanh( K/bm + b/T)


with K = (J/T)^1/2. The analysis of this equation for various temperatures T and couplings J, b reveals a phase transition at the critical temperature T_c = J. Introducing the dimensionless parameter t=(T/T_c-1) and expanding (<ref>) in small couplings the famous power law dependence on temperature for the magnetisation emerges:


    m ≃√(3) (K-1)^1/2/K^3/2   ∼ |t|^1/2


Here we recognise one of the very typical power laws which are ubiquitous to critical systems.
The quantity we are most interested in though is the second derivative of the free energy F wrt. b, which is basically the 2-point correlation function of the spins s_i. Again, expanding the second derivative of the free energy in small couplings and looking in the neighbourhood of the critical temperature T_c yields 


    ⟨ s_i,s_j⟩∼b^2/T_c|t|^-1


again displaying power law behavior with a power coefficient γ = 1. The innocent looking equation <ref> has actually tremendous consequences, as it implies that correlation does not simply restrict to nearest neighbours but goes over very long distances only slowly decaying; further, because of the power law behavior, there will be self-similar, fractal patterns in the system: islands of equal magnetisation will form within other islands and so on, all the way through all scales.
Also, the correlation diverges at the criticality point T_c. We will carry out the explicit calculations for our case of interest - non-constant matrix couplings - later one, in section <ref>.



 §.§ Criticality in real-world networks


Two of the main motivations why we look for criticality and exploit on it in artificial networks, are the universal arising of this phenomenon as well as various hints of its occurrence in biological <cit.> and neural systems <cit.>; once systems get "sizable" enough, gaining complexity, critical behavior emerges, which also applies to man-made nets <cit.>. Various measures can be formulated to detect criticality, and they all show power law distribution behavior. In the world wide web, e.g. the number of links to a source, and the number of links away from a source, both exhibit power law distribution


    P(k) ∼ k^-γ


for some power coefficient γ≠ 0. Similar behavior can be uncovered in various other networks, if sizable enough, such as citation behavior of scientific articles, social networks, etc. A simple, generic metric to detect criticality in networks is the degree distribution, defined as the number of (weighted) links connecting to one node. 

Further, also the correlation between nodes is non-trivial, such that nodes with similar degree have higher probability of being connected than nodes of different degree <cit.>, chapter VII. We will follow a similar path as proposed above and grow an experimental network with new nodes having the simplest preferential and directed attachment towards existing nodes, as a function of their degree:


    Π(k) ∼ k^α


Here, Π(k) denotes the probability that some node will grow a link to another node of degree k. Hence, every new node, will prefer nodes with higher degrees, leading to the overall power distribution observed in the real world systems.
Additional metrics we look at are single neuron activity as well as layer activity and pattern behavior; more details on that in section <ref>. 



§ CRITICALITY IN DEEP LEARNING NETS





 §.§ From feed-forward to fully connected architecture



We will focus now on a feed-forward network, with two layers, a_i and b_j connected via a weight matrix w_ij; In order to probe our system for criticality, we write down its Hamiltonian



    H =  -1/2N∑_ij^N w_ija_ib_j 
        - h ∑_i^N b_i


which has been first formulated in the seminal paper <cit.>. Here, the values of the a and b
are {0,1}. Further, by absorbing the biases b_i in the usual way we can assume our weight matrix has the form: 


    W = [  2Nh    0    ⋯    0;  2Nh w_11    ⋯ w_1n;    ⋮    ⋮    ⋱    ⋮; -2Nh w_n1    ⋯ w_nn ]


while the V_i read (1,V_1,⋯, V_N).

This Hamiltonian describes a two layer net containing rectified linear units (ReLU) in the b-layer with a common bias term h. The weight matrix w_ij sums the binary inputs coming from the a_i and those are fed into b_i; depending whether the ReLU threshold has been reached, a_i is activated, hence the binary values allowed for both, inputs and b-layer.

Further, we show in appendix <ref>, that the partition function is up to a constant the same for the units taking values in {± 1} or {0,1}. By redefining N+1 → N We can then formulate the partition function as


    Z =  ∑_a,b∈{± 1}  e^-β/2N∑_ij W_ija_ib_j


where β is the inverse temperature 1/T. This is the partition function of a bipartite graph with non-constant connection matrix w.

However, it turns out, that the partition function of the fully connected layer is the highest contribution (1st order) of our feed forward network (see appendix <ref>), hence further simplifying the expression to


    Z =  ∑_s_i∈{± 1}  e^-β/2N∑_ij W_ijs_is_j



We will now proceed and compute the free energy F, defined as F = -TlnZ, using the procedure presented in <cit.>. From the free energy we
then find all quantities of interest, especially the 2-point correlation function of the neurons.



 §.§ Fully connected architecture with non-constant weights


In order to solve the CW model analytically, one has to perform the sum over spins, which is hindered by the quadratic term s_is_j. The standard way to overcome this problem is the gaussian linearisation trick which replaces quadratic term by its square root - linear in s_i  and one additional continuous variable - the "mean" field, which is being integrated over entire R:


    e^a^2 = 1/√(2π)∫_-∞^∞ dx e^-x^2/2 + √(2)ax


which in physics, is known as the Hubbard–Stratonovich transform.

Unfortunately our coupling is not scalar, and hence we will linearise the sum term by term to keep track of all the weight matrix entries.
First we will insert N identities via the Dirac delta function into our Hamiltonian as used in  (<ref>):


    H(s)    =  -1/2N∑_ij^N s_iW_ijs_j 
       = -1/2N∏_k∫_-∞^∞ dV_kδ(s_k-V_k)∑_ij^N V_iW_ijV_j 
       = ∏_k∫_-∞^∞ dV_kδ(s_k-V_k) H(V)


With the definition of the delta function 
δ(x) = 1/2π i∫_-i∞^i∞dye^xy
the partition function (<ref>) reads now


    Z(s) =  ∏_k∫_-∞^∞ dV_kδ(s_k-V_k)
    ∑_s_i∈{± 1}e^-β H(V)
       ∼∏_k∫_-∞^∞ dV_k∫_-i∞^i∞dU_k∑_s_i∈{± 1} e^U_k(s_k-V_k)
    e^-β H(V)
       =  ∏_k∫_-∞^∞ dV_k∫_-i∞^i∞dU_k e^-U_kV_k + ln(cosh U_k)
    e^-β H(V)


As already stated, we could perform the sum over the binary units s_i, since they show up linearly in the exponential after the change of variables via delta identity[In general we're not interested in numerical multiplicative constants, as later on, when logging the partition and computing the free energy, those terms will be simple additive constants without any contribution after differentiating the expression]; we effectively converted the sum over binary values {± 1} into integrals over R, leading to


    Z =  c ∏_i ∫_-∞^∞dV_i ∫_-i∞^i∞dU_i   e^-H^g(V,U,T)


with a generalised Hamiltonian


    H^g =     -β/2N∑_ij W_ijV_iV_j +∑_i[ U_iV_i - ln(coshU_i)]
        =    -β/2N∑_ij w_ijV_iV_j - β h∑_iV_i
       +∑_i[ U_iV_i -ln(coshU_i)]



Ultimately we are interested in the free energy per unit, which contains the partition function, via


    F = lim_N→∞ (-TlnZ)/N



From F we can now obtain all quantities of interest via derivatives, in our case with respect to h. The partition function Z still contains a product of double integrals, which can be solved via the saddle point approximation; we recall here the one-dimensional case 


    ∫_-∞^∞ dx e^-f(x)≈(2π/f”(x_0))^1/2e^-f(x_0)


where x_0 is the stationary value of f and f”(x_0) is in our case the Hessian evaluated at the stationary point:


    H^g_V_iV_j   =  -β/Nw_ij
    
    H^g_U_iU_j   =  -δ_ij(1-tanh^2U_i)
    
    H^g_V_iU_j   =  δ_ij


while H^g is given in (<ref>).


The expression <ref> can now be computed by applying simultaneously the saddle point conditions for both integrals. The stationarity conditions[We keep in mind that we enlarged W to contain h as well, hence the explicit equations are h dependent] for V_i and U_i give


    ∂ H^g/∂ V_i    = -β/N∑_jW_ijV_j + U_i=  0
        = -β(∑_jw_ijV_j/N + h) + U_i=  0
    ∂ H^g/∂ U_i    = V_i - tanhU_i = 0


which combined deliver the self consistency mean field equation of the fully connected layer (<ref>). Further, denoting H^g_0 the the Hamiltonian satisfying the stationarity conditions, it reads


    H^g_0 =    β/2N∑_ij w_ijV_iV_j 
    
    -   ∑_ilncoshβ(∑_jw_ijV_j/N + h)


Equation (<ref>) already displays manifestly the consistency equation for the mean field, as taking the first derivative wrt. V_i leaves exactly the consistency equation over per its construction;

Now we can rewrite the free energy (<ref>) as


    F    = lim_N→∞T/N[H^g_0 + ln H^g_hh]∼lim_N→∞
       [1/2N^2∑_ij w_ijV_iV_j
    +1/N^2∑_ijln[ w_ij(1-V^2_i)-1] 
       -  T/N∑_ilncoshβ(∑_jw_ijV_j/N + h)]


We need to address now the large N limit; obviously
the second term coming from the determinant clearly vanishes in the large-N limit, as the logarithm is slowly increasing, while we divide through N^2;
the first term - a double sum over V_i is of order N^2 and hence a well defined average in the limit;
the last term - lncosh, when expanded, is again linear in the sum[The interior sum over j is an  average, hence well defined in the limit; after expansion, we're left with the outer sum (over i), which is again a well defined average when divided by N ], and hence a well defined average after dividing through N, hence we're left with the free energy


    F =   T/NH_0^g
    
    =   1/2N^2∑_ij w_ijV_iV_j
       -  T/N∑_ilncoshβ(∑_jw_ijV_j/N + h)]


We're at the point now, where all quantities of interest can be derived from the free energy F; the order parameter (aka magnetisation when dealing with spins) per unit is defined as 


    m    ≡dF/dh = .∂ F/∂ h|_V^st + 0∂ F/∂ V_i.∂ V_i/∂ h|_V^st


The second term on the right vanishes identically, as we recognize it being evaluated at the stationarity condition V^st for the Hamiltonian. The contribution of the first term is:


    ∑_iw_ikV_k/N    = 1/N∑_itanhβ
    (∑_kw_ikV_k/N + h)w_ik
       ⇕
    
    V_i    = tanhβ
    (∑_kw_ikV_k/N + h)


which is (the weighted sum version of) the iconic self-consistency mean field equation of the CW magnet (<ref>).

The critical point, P_c is located where the correlation function diverges for h→ 0; the 2-point correlation function (aka susceptibility when dealing with spins) is the second derivative of F, i.e. the derivative of (<ref>) wrt. h:


    P_c    ≡d^2F/dh^2 = dm/dh
       ⇕
    ∂ V_i/∂ h   = β(
    1-V_i^2)(1+ ∑_kw_ik∂ V_k/∂ h/N)


where we used the original equation (<ref>) for taking the derivatives. 
It is worth contemplating first equations (<ref>) and (<ref>). They both capture the essence of the criticality of our system, including it's power law behavior. When the weight matrix reduces to a scalar coupling, both equations reduce to the classical CW system and display the behavior shown in (<ref>) and (<ref>). Furthermore, eq. (<ref>) encodes all the information needed for finding the critical point of matrix system at hand; we recall that all Vs (and their derivatives) are already implicitly "solved" in terms of h and w_ij via the stationarity equation (<ref>) and hence the V_i
are just place holders for functions of w and h; 
we're thus left with a non-linear system of first order differential equations in N variables, which
will produce poles for specific values of the couplings and temperature at criticality.  





§ EXPERIMENTAL RESULTS





After investigating criticality through the partition function in our theoretical setup, now we turn to a practical question: do current deep learning networks exhibit critical behaviour, or put it differently, can we spot traces of critical phenomena in them? Instead of directly attacking the partition function of real world deep neural nets, we start with the practical observation, that systems at around criticality show off power law distributions in certain internal attributes. 




Concretely for networks <cit.> we look for traces of power laws in weight distributions, layer activation pattern frequencies, single node activation frequencies and average layer activations.
In the following we will present experimental results for multilayer feed-forward networks, convolutional neural nets and autoencoders. 



For all networks we ran experiments on the CIFAR-10 dataset, training each models for 200 epochs using ReLU activations and Adam Optimizer without gradient clipping and run inferences for 100 epochs. The feed forward network had 3 layers with 500, 400 and 200 nodes, the CNN had 3 convolutional layers followed by 3 fully connected layers and the autoencoder had one layer with 500 nodes. 

For weight distributions we looked at sums of absolute values of the outgoing weights at each node, as a weighted order of the node. In fig. <ref> we have a log-log plot of counts versus the node order as defined above, and detect no linear behavior.



For layer activation patterns we counted the frequency of each layer activations through the inference epochs. Figures <ref> and <ref> are log-log plots of layer activation frequencies versus their respective counts for the feed-forward layer the autoencoder. As we see, the hidden layer activation pattern frequencies of the Autoencoder resembles a truncated straight line, indicating that learning hidden features in unsupervised manner can give rise for scale free, power law phenomena in accordance with the findings of <cit.>, but no other architectures show traces of any power law. 

For single node activation frequencies we counted the frequency of each node activations through the inference epochs. 



Figures <ref> and <ref> depict the behavior of feed-forward and CN network. The flat, nearly horizontal line in the latter architecture is again a sign of missing exponent whatsoever. 

As a last measure we employed the sum of activations defined as the average activations on each layer throughout the inference epochs. 





Spontaneous and detectable criticality did not arise in classical architectures so the next step will be to create and experiment with systems that have induced criticality and learning rules that take into account criticality. Our first approach was to grow a fully connected net using the preferential attachment algorithm to induce at least some power law in node weights, and use the fully connected net as a hidden to hidden module. We further experimented with different solutions, regarding input and read out of activations from this hidden to hidden module, without changing the power law distribution. (This would simulate a system located at a critical state, with power law weight distribution). Our findings so far show that learning in these systems is very unstable without any advancement in learning and inference. The fundamental missing part is how to naturally induce a critical state in a network, which is equipped with learning rules that inherently take into account the critical state. For that we need new architectures and new learning rules, derived from the critical point equations (<ref>).   




§ SUMMARY AND OUTLOOK


Summary: In this article we make our first steps in investigating the relationship between criticality and deep learning networks. After a short introduction of criticality in statistical physics and real world networks we started with the theoretical setup of a fully connected layer. We used continuous mean field approximation techniques to tackle the partition function of the system ending up with a system of differential equations that determine the critical behaviour of the system. These equations can be the starting point for a possible network architecture with induced criticality and learning rules exploiting criticality. After that we presented results of experiments aiming to find traces of power law distributions in current deep learning networks such as multilayer feed-forward nets, convolutional networks and autoencoders. The results - except for the autoencoder - were affirmative in the negative sense, setting up as next the necessity to create networks with induced criticality and learning rules that exploit the critical state. 

Outlook: Obviously the fully connected layer, which can be solved analytically on the theoretical side is of limited importance, as it translates into a rather simplistic architecture; more realistic, widely used set-ups, e.g. convolutional or recurrent nets, do very well contain the feed-forward mechanism, but are strongly deviating  and hence only partially mapped to our theoretical treatment; it would definitely be essential to address theoretically the convolution mechanism of deep nets and establish a link between the theoretical and experimental side; also inducing criticality into the net via eq. (<ref>) could prove beneficial and might very well affect learning behavior and flow on the surface on the loss function.




Appendix 




§ DIFFERENT UNIT VALUES


We here show that the partition function with Hamiltonian 

    H_{0,1} = ∑_ija_iw_ija_j + h_ia_i

who's units are taking values in {0,1} has the same qualities as encoded in the partition function with Hamiltonian H_{± 1}, who's units take values in {± 1}.


We rewrite the Hamiltonian in (<ref>) with units taking values in {± 1} (using Einstein's summation convention over double indices) :


    H = 1/4(1-u_i)w_ij(1-u_j) + 1/2h_i(1-u_i)


where the u_i and v_i take values in {± 1}. Carrying now the multiplications in (<ref>) yields


    H    = 1/4∑_ijw_ij
    + 1/4u_iw_iju_j
       - 2/4u_iw_ij1_j 
    - 1/2h_iu_i + 1/2h_i
       = 1/2(c + u_iw_iju_j 
    + h'_i u_i)


with h_i' = -w_ij1_j - h_i. Hence when computing the partition Z with (<ref>) we obtain


    Z = ∑_u∈{± 1}e^H = e^c∑_a∈{0,1}
    e^ a_iw_ija_j + h'_ia_i


where the right hand side is the original Hamiltonian with a shifted coupling h'. The additional constant c factors out completely and hence when taking the logarithm and the second derivative it won't change the outcome. Also we note that the second derivative wrt. h' is ∂_h'h' = ∂_hh.



§ FIRST ORDER CONTRIBUTION



We consider here the Hamiltonian of the bi-partite graph connected via  weight matrix w (with Einstein summation convention):


    H_b = u_iw_ijv_j


with the free energy 


    F_b = -ln∑_u,v∈{± 1}exp( u_iw_ijv_j )


Without any loss of generality we set the temperature T=1, and we won't keep track of it. 
Carrying the partial sum over v_i yields


    F_b    = -ln∑_u∏_j[exp(u_iw_ij) + exp(-u_iw_ij)]
       =-ln∑_u∏_j[2cosh(u_iw_ij) ]


The sum over the v_i is understood as a collection of 2^N terms, each corresponding to a unique combination of 0's and 1's in the vector of length N representing that specific state of the spins; however, the sum can be conveniently written as a product of N binary summands, where each contains exactly the two possible states of the ith spin - this is where the product over j comes from in upper formula.
Expanding now to lowest order in w we obtain


    F_b    ∼  -ln∑_u∏_j(1 + (u_iw_ij)^2/2 )
       ∼ -ln∑_uexp∑_j(u_iw_ij)^2/2
       =-ln∑_u e^H(u_i)


where H(u_i) is the Hamiltonian of the fully connected graph, defined as (Einstein summation convention) 


    H(u_i)    = ∑_j(u_iw_ij)^2/2
       =1/2∑_ik(∑_j w_ijw_jk)_w'_iku_iu_k
       =1/2∑_iku_iw'_iku_k


A few notes are in place regarding eq. (<ref>): the matrix w'_ik is now symmetric by construction and hence mediates between equally sized (actually identical) layers; further, all higher terms of the cosh function are even, hence all contributions are higher order, symmetric interactions of the layer u_i with itself.





9
per_bak_nature
Per Bak,
"How Nature Works: the science of self-organized criticality", Copernicus Springer-Verlag, New York, 1996 

Scale_Invariance_Lesne_Lagues
Lesne Annick, Lagues Michel,
"Scale Invariance,
From Phase Transitions to Turbulence", Springer, 2012

pruessner
Gunnar Pruessner, "Self-organised criticality", Cambridge University Press, 2012

bio_sys_poised_crit
Thierry Mora, William Bialek,
"Are biological systems poised at criticality?",	
arXiv:1012.2242 [q-bio.QM]


Cover_Joy_Elements_of_Information_Theory
Thomas M. Cover, Joy A. Thomas,
"Elements of Information Theory", Wiley Series, 2006

fukushima_self_org
Kunihiko Fukushima,
"Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition
Unaffected by Shift in Position"

<www.cs.princeton.edu/courses/archive/spr08/cos598B/Readings/Fukushima1980.pdf>


barra_phase_boltzmann
Adriano Barra,Giuseppe Genovese, Peter Sollich, Daniele Tantari,
"Phase transitions in Restricted Boltzmann Machines with generic priors", arXiv:1612.03132 [cond-mat.dis-nn]

hopf_boltzmann  
Adriano Barra, Alberto Bernacchia, Enrica Santucci, Pierluigi Contucci, 
"On the equivalence of Hopfield Networks and Boltzmann Machines", 	arXiv:1105.2790 [cond-mat.dis-nn]

hopfield_nn_phys_sys 
J. J. Hopfield, "Neural Networks and physical systems with emergent collective
computational abilities," Proceedings of the National Academy of
Science, USA, 79 (1982) 2554-2558.

peterson_anderson_mft
Carsten Peterson, James R . Anderson,
"A Mean Field Theory Learning Algorithm for Neural
Networks", Complex Systems 1 (1987) 995- 1019

loss_surface
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gerard Ben Arous, Yann LeCun,
"The Loss Surfaces of Multilayer Networks",
arXiv:1412.0233 [cs.LG]

spinglass_bialek
Gasper Tkacik, Elad Schneidman, Michael J. Berry II, William Bialek,
"Spin glass models for a network of real neurons", arXiv:0912.5409 [q-bio.NC]

energyae 
Hanna Kamyshanska, Roland Memisevic,
"The Potential Energy of an Autoencoder" 

<https://www.iro.umontreal.ca/ memisevr/pubs/AEenergy.pdf>
 


complex_crit
Kim Christensen, Nicholas R. Moloney,
"Complexity and Criticality (Advanced Physics Texts)",
Imperial College Press, 2005

life_order
Philip Ball,
"One rule of life: Are we poised on the border of order?", New Scientist, April 2014

renorm_stat_phys
Uwe C. Tauber,
"Renormalization Group: Applications in Statistical Physics", Nuclear Physics B, (2011) 1–28

origin_order
Stuart A. Kauffman, "The Origins of Order: Self-Organization and Selection in Evolution", Oxford University Press, 1993

scale_renorm_univers
H.E. Stanley,
"Scaling,universality and renormalization:three pillars of modern critical phenomena", 

<http://journals.aps.org/rmp/abstract/10.1103/RevModPhys.71.S358>


salinas 
Silvio Salinas,
"Introduction to statistical physics", Springer, 2001

cw_full_solution 
Martin Kochmanski, Tadeusz Paszkiewicz, Slawomir Wolski,
"Curie-Weiss magnet: a simple model of phase transition",	arXiv:1301.2141 [cond-mat.stat-mech]

crit_brain_dynam
Dante R. Chialvo,
"Critical brain dynamics at large scale", 	arXiv:1210.3632 [q-bio.NC]


weak_pair_wise_corr_popul
Elad Schneidman, Michael J. Berry II, Ronen Segev, William Bialek,
"Weak pairwise correlations imply strongly correlated network states in a neural population", 	arXiv:q-bio/0512013 [q-bio.NC]


stat_mech_complex_nets
Reka Albert, Albert-Laszlo Barabasi,
"Statistical mechanics of complex networks",	arXiv:cond-mat/0106096 [cond-mat.stat-mech]

 
zipf_crit_no_fine
David J. Schwab, Ilya Nemenman, Pankaj Mehta,
"Zipfs law and criticality in multivariate data without fine-tuning", arXiv:1310.0448v3 [q-bio.NC]

crit_brain_dynam
Dante R. Chialvo,
"Critical brain dynamics at large scale", 	arXiv:1210.3632 [q-bio.NC]





