Chaos-guided Input Structuring for Improved Learning in Recurrent Neural Networks
Priyadarshini Panda Kaushik Roy
q-bio.NC cs.NE physics.bio-ph


Kinematics of CMEs and associated shock waves as deduced from LASCO data: comparative analysis
    Zagainova  Yu.S.
    
==============================================================================================





empty




§ INTRODUCTION

Brain actively untangles the input sensory data and fits them in behaviorally relevant dimensions that enables an organism to perform recognition effortlessly, in spite of variations <cit.>. For instance, in visual data, object translation, rotation, lighting changes and so forth cause complex nonlinear changes in the original input space. However, the brain still extracts high-level behaviorally relevant constructs from these varying input conditions and recognizes the objects accurately. What remains unknown is how brain accomplishes this untangling. 

Here, we introduce the concept of chaos-guided input structuring in a reservoir computing network that provides an avenue to untangle stimuli in the input space and improve the ability of a stimulus to entrain neural dynamics. Specifically, we show that the complex dynamics arising from the recurrent structure of a randomly connected reservoir <cit.> can be used to extract an explicit phase relationship between the input stimulus and the spontaneous chaotic neuronal response. Then, aligning the input phase along the dominant projections determining the intrinsic chaotic activity, causes the random chaotic fluctuations or trajectories of the network to become locally stable channels or dynamic attractor states that, in turn, improve its' inference capability. In fact, using mean field analysis, we derive the effect of introducing varying phase association between the input and the network's spontaneous activity. Our results demonstrate that successful formation of stable attractors is strongly determined from the input alignment.
 


§ MODEL DESCRIPTION

We describe the effect of chaos guided input structuring on a standard firing-rate based reservoir model of N interconnected neurons. Specifically, each neuron in the network is described by an activation variable x_i ∀ i= 1,2,...N, satisfying

    τ dx_i/dt = -x_i + ∑_j=1^N W_ij r_j + W_InputI;
    z = ∑_j=1^N W_Out r_j

where r_i(t) = ϕ(x_i(t)) represents the firing rate of each neuron characterized by the nonlinear response function, ϕ(x) = tanh(x) and τ =10 ms is the neuron time constant. W represents a sparse N× N recurrent weight matrix (with W_ij equal to the strength of the synapse connecting unit j to unit i) chosen randomly and independently from a Gaussian distribution with 0 mean and variance, g^2/p_c N <cit.>, where g is the synaptic gain parameter and p_c is the connection probability between units. 
The output unit z reads out the activity of the network through the connectivity matrix, W_Out, with initial values drawn from a Gaussian distribution with 0 mean and variance 1/N. The readout weights are trained using Recursive Least Square (RLS) algorithm <cit.>. The input weight matrix, W_Input, is drawn from a Gaussian distribution with zero mean and unit variance. The external input, I, is an oscillatory sinusoidal signal, I = I_0 cos(2π ft + χ), with amplitude I_0, frequency f, that is the same for each unit i. Here, we use a phase factor χ chosen randomly and independently from a uniform distribution between 0 and 2π. This ensures that the spatial pattern of input is not correlated with the recurrent connectivity initially. Through input alignment analysis, we then obtain the optimal phases to project the inputs in the preferred direction of the network's spontaneous activity. In all our simulations (without loss of generality), throughout the paper we have assumed, p_c = 0.1, N=800, g=1.5, f= 10 Hz. 






§ SUBSPACE ALIGNMENT

First, we ask the question how is the subspace of input driven activity aligned with respect to the subspace of spontaneous activity of a recurrent network. Using Principal Component Analysis, we observed that the input-driven trajectory converges to a uniform shape becoming more circular with increasing input amplitude <cit.>. We utilize the concept of principal angles, introduced in <cit.>, to visualize the relationship between the chaotic and input driven (circular) subspace. Specifically, for two subspaces of dimension D_1 and D_2 defined by unit principal component vectors (that are mutually orthogonal) V_1^a, for a = 1,2,...D_1 and V_2^b, for b = 1,2,...D_2, the angle between them is 

    θ = arccos(min(SingularValueOf(V_1^a . V_2^b)))


Fig. <ref> (a) schematically represents the angle between the circular input driven network activity and the irregular spontaneous chaotic activity. Here, θ_chaos (and θ_driven) refers to the subspace defined by the first two Principal Components (PCs) of the intrinsic chaotic activity (and periodic driven activity). It is evident that rotating the circular orbit by θ_rotate will align it along the chaotic trajectory projection. 
We observe that aligning the inputs in directions (along dominant PCs) that account for maximal variance in the chaotic spontaneous activity facilitates intrinsic noise suppression at relatively low input amplitudes, thereby, allowing the network to produce stable trajectories. For instance, instead of using random phase input, we set I = I_0 cos(2π ft + Θ) and visualize the network activity as shown in Fig. <ref> (b). Even at lower amplitude of I_0 =1.5, we observe a uniform circular orbit (in the PC subspace) for the network activity that is characteristic of reduction in intrinsic noise and input sensitization. In fact, even after the input is turned off after t=50 ms, the neural units yield stable and synchronized trajectories with minimal variation across different trials (Fig. <ref> (b, Right)) in comparison to the random phase input driven network (of higher amplitude) in <cit.>. This shows the effectiveness of subspace alignment for intrinsic noise suppression. In addition, working in low input-amplitude regimes offers an additional advantage of higher network dimensionality <cit.>, that in turn improves the overall discriminative ability of the network. Note, previous work <cit.> have shown that spatial structure of the input does not have a keen influence on the spatial structure of the network response. Here, we bring in this association explicitly with subspace alignment. 

Θ, in the above analysis, is the input phase that corresponds to a subspace rotation of driven activity toward spontaneous chaotic activity. We observe that the temporal phase of the input contributes to the neuronal activity in a recurrent network. Fig. <ref> (c) illustrates this correlation wherein the input phase determines the orientation of the input-driven circular orbit with respect to the dominant subspace of intrinsic chaotic activity. For a given input frequency (f =10 Hz), input phase, Θ = 83.2^∘, aligns the driven activity (θ_driven) along the chaotic activity (θ_chaos) resulting in θ_rotate =0^∘ for varying input amplitude (I_0 = 1.5, 3). 
An interesting observation here is that the frequency of the input modifies the orientation of the evoked response that yields different input phases at which θ_chaos and θ_driven are aligned (refer to Fig. <ref> (c, Right)). We also observe that the subspace alignment is extremely sensitive toward the input phase in certain regions with abrupt jumps and non-smooth correlation. This non-linear behavior is a consequence of the recurrent connectivity that overall shapes the complex interaction between the driving input and the intrinsic dynamics. While this correlation yields several important implications for network modeling experiments, we utilize this behavior for subspace alignment. Consequently, in all our experiments, for a given  θ_rotate, we find a corresponding input phase Θ that approximately aligns the input in the preferred direction. 



§ IMPACT OF INPUT STRUCTURING ON DISCRIMINATIVE CAPABILITY

Next, we describe the implication of input alignment along the chaotic projections on the overall learning ability of the network. 
First, we trained a recurrent network with two output units to generate a timed response at t= 1s as shown in Fig. <ref> (d, Right). Two distinct and brief sinusoidal inputs (of 50 ms duration and amplitude I_0 = 1.5) were used to stimulate the recurrent network. The network trajectories produced were then mapped to the output units using RLS training (to learn the weights W_Out). Here, the network (after readout training) is expected to produce timed output dynamics at readout unit 1 or 2 in response to input I_1 or I_2, respectively. The network is reliable if it generates consistent response at the readout units across repeated presentations of the inputs during testing, across different trials. This simple experiment utilizes the fact that neural dynamics in a recurrent network implicitly encode timing that is fundamental to the processing and generation of complex spatio-temporal patterns. Note, in such cases of multiple inputs, values of both inputs are zero, except for a timing window during which one input is briefly turned on in a given trial. 






Since both the inputs, in the above experiment, have same amplitude and frequency dynamics, the circular orbit describing the network activity in the input-driven state (for both inputs) is almost similar giving rise to one principal angle (θ_driven_1,2 in Fig. <ref> (e)) for the input subspace. To discriminate between the output responses for the two inputs, it is apparent that the inputs have to be aligned in different directions. One obvious choice is to align each input along two different principal angles defining the chaotic spontaneous activity (i.e. I_1 along ∠ PC1.PC2 and I_2 along ∠ PC3.PC4). Note, ∠ PC1.PC2 denotes the angle θ calculated using Eqn. <ref>. Another approach is to align I_1 along ∠ PC1.PC2 ≡θ_chaos and I_2 along ∠ PC1.PC2 + 90^∘≡θ_chaos,90^∘ as shown in Fig. <ref> (e). We analyze the latter in detail as it involves input phase rotation in one subspace that makes it easier for formal theoretical analysis. 

To characterize the discriminative performance of the network, we evaluated the Euclidean distances (measured as √(1/N∑_i=1^N(r_i,1(t)-r_i,2(t))^2) , where r_1(t) (r_2(t)) is the firing rate activity of the network corresponding to I_1 (I_2)) between the inter-/intra-input trajectories in response to different inputs (I_1, I_2), and to a slightly varied version of the same input (for instance, I_1,2 = (I_0 +ϵ)cos(2π ft + Θ_1,2) where ϵ is a random number between [0, 0.5]) and Θ_1 (Θ_2) is the input phase that aligns I_1 (I_2) along θ_chaos (θ_chaos, 90^∘). The inter-/intra-input trajectory distances are plotted in Fig. <ref> (f) for both scenarios-with and without input alignment. It is desirable to have larger inter-trajectory distance and small intra-trajectory distance such that the network easily distinguishes between two inputs while being able to reproduce the required output response even when a particular input is slightly perturbed. We observe that aligning the inputs in direction parallel and perpendicular to the dominant projections (Fig. <ref> (f, Middle)) increases the inter-trajectory distance compared to the non-aligned case (Fig. <ref> (f, Top)) while decreasing the intra-input trajectory separation. This further ascertains the fact that subspace alignment reduces intrinsic fluctuations within a network thereby enhancing its prediction capability. Note, without input alignment, the intrinsic fluctuations cannot be overcome with low-amplitude inputs (I_0=1.5). Hence, for fair comparison and to obtain stable readout-trainable trajectory in the non-aligned case, we use a higher input amplitude of I_0 =3.

We hypothesize that intrinsic noise suppression occurs as input subspace alignment along dominant projections (that account for maximal variance such as PC1, PC2) causes chaotic trajectories along different directions (in this case, along θ_chaos, θ_chaos,90^∘)  to become locally stable channels or attractor states. These attractors behave as potential wells (or local minima from an optimization standpoint) toward which the network activity converges for different inputs. Thus, the successful formation of stable yet distinctive attractors for different inputs are strongly influenced by the orientation along which the inputs are aligned. As a consequence of our hypothesis, depending upon the orientation of the input with respect to the dominant chaotic activity (θ_chaos in Fig. <ref> (e)), the extent of noise suppression will vary, for a particular trajectory, that will eventually alter the stability of the attractor states. To test this, we rotated I_2 (from θ_chaos,90^∘) further by 90^∘ (θ_chaos,180^∘ in Fig. <ref> (e)) and monitored the intra-trajectory distance. Note, I_1 and I_2 are anti-phase correlated in the chaotic subspace. In Fig. <ref> (f, Middle) corresponding to 90^∘ phase difference between (I_1, I_2), I_2 corresponds to a more stable attractor than I_1 since the intra-distance for the former is lower. In contrast, in Fig. <ref> (f, Right) corresponding to 180^∘ phase difference, I_1 turns out be more stable than I_2. Note, the 90^∘, 180^∘ phase difference between I_1, I_2 (mentioned above and in the remainder of the paper) refers to the phase difference between the inputs in the chaotic subspace after subspace alignment using Θ. For our analysis, Θ_1 = 83.2^∘, Θ_2 = 111^∘ yields ∼ 90^∘ phase between I_1, I_2 in chaotic subspace, while Θ_1 = 83.2^∘, Θ_2 = 263.2^∘ yields ∼ 180^∘ phase. 

In addition to the trajectory distance, visualizing the network activity in the 3-D PC space (Fig. <ref> (g)), also, shows the influence of input orientation (and hence the phase correlation) toward formation of distinct attractor states. Since I_1, I_2 are aligned in the subspace defined by ∠ PC1.PC2, the 2D projection of the circular orbit onto PC1 and PC2 in both input aligned scenarios (90^∘, 180^∘ phase) are comparable. However, the third dimension, PC3, marks the difference between the two input projections. In fact, the progress of the network activity as time evolves (shown by dashed arrows in Fig. <ref> (g)) follows a completely different cycle for the input aligned scenarios. The change in the overall rotation cycle from anti-clockwise (I_2 with 90^∘ phase, Fig. <ref> (g, Middle)) to clockwise (I_2 with 180^∘ phase, Fig. <ref> (g, Bottom)) can be viewed as an indication toward the altering of the attractor state stability. On the other hand, the non-aligned case with I_0 =3 yields incoherent and more random trajectory (Fig. <ref> (g, Top) representative of intrinsic noise. In order to get more coherent activity and to suppress the noise further, we need to increase the input amplitude to I_0 ≥ 5 as shown in <cit.>.



§ MEAN FIELD ANALYSIS

To explain the above results analytically, we use mean-field methods developed to evaluate the properties of random network models in the limit N →∞ <cit.>. A key quantity in Mean Field Theory (MFT) is the average autocorrelation function that characterizes the interaction within the network as

    C(τ) = 1/N ∑_i=1^N <ϕ (x_i(t)) ϕ (x_i(t+τ))>

where <> denotes the time average. The main idea of MFT is to replace the network interaction term in Eqn. 1 by Gaussian noise η such that dx_i^1/dt = -x_i^1 +η, where x_i = x_i^0 + x_i^1 and x_i^0 (t) = A cos(2π ft + ζ) with A = I_0/√(1+(2π ft)^2). Here, ζ incorporates the averaged temporal phase relationship between the reservoir neurons and the input induced by input subspace alignment, ζ(θ_rotate) = Θ. The temporal correlation of η is calculated self-consistently from C(τ).  For self-consistence, the first and second moment of η must match the moments of the network interaction term. Thus, we get <η_i (t)> = 0 as mean of the recurrent synaptic matrix <W_ij> = 0. For calculating the second moment , we use the identity <W_ijW_kl> = g^2 δ_ijδ_kl/N and obtain <η_i (t) η_j(t+τ)> = g^2C(τ). Combining this result with the MFT noise-interaction based network equation yields

    d^2 Δ (τ)/d τ^2 = Δ(τ) - g^2C(τ)

where Δ(τ) = < x_i^1 (t) x_i^1 (t+τ)>. Eqn. <ref> resembles the Newtonian motion equation of a classical particle moving under the influence of force given by the right hand side of the equation. This force depends on C that, in turn, depends on the input subspace alignment (ζ) which directs the initial position of the particle (or state of the network Δ(0)). From this analogy, it is evident that analyzing the overall potential energy function of the particle (or network) will be equivalent to visualizing the different attractor states formed in a network in response to a particular input stimulus. Thus, we formulated an expression for the correlation function (with certain constraints) using Taylor series expansion, that allows us to derive the force and hence the dynamics of the network under various input alignment conditions. 

The non-linear firing rate function r(x) = ϕ(x) = tanh(gx) can be expanded with Taylor series for small values of g, i.e. g = 1 + δ, where δ denotes a small increment in g beyond 1. Note, g=1+δ satisfies the criterion, g >1 <cit.>, to operate the networks in chaotic regime. Also, the overall network statistics does not change with g being expressed as a gain factor in the firing-rate function instead of overall synaptic strength. Using tanh(gx) ≃ gx-1/3g^3x^3 +2/15g^5x^5, we can express C(τ) from Eqn. <ref> as  C = 1/2g^2A^2cos(ωτ +2ζ) + l(g^2-2g^4m) + 2/3g^6l^3, where m=Δ(0), l=Δ(τ), ω = 2π f. Now, we can express Eqn. <ref> as d^2l/d τ^2 = l- C. Writing l=kδ due to the small limit of g, Eqn. <ref> simplifies to

    d^2k/dt^2 = -G +  n k -2/3k^3


where G = g^2A^2cos(ωτ +2ζ)/(2δ^3) and n is a parameter defined in terms of m, δ. Note, supplementary section <cit.> provides a detailed derivation of Eqn. <ref> and comments about the assumptions on initial conditions. Note, Eqn. <ref> is an approximate version of Eqn. <ref> that depicts network activity in the manner of Newtonian motion independent of all intrinsic time (or averaging parameters) while taking into account the influence of input alignment. Now, we can express the potential of the network driven by a force, F, equivalent to the right hand side of Eqn. <ref> as

    V= -∫Fdk = Gk-n/2k^2+k^4/6

We solve Eqn. <ref>, <ref> with initial conditions k(0) =1, k̇(0) =0 and monitor the change in force, F, and potential, V, for different values of G. First, let us examine the attractor state formation when there is no input stimulus (i.e. G=0) by visualizing the potential V. For G=0, the expressions for force and potential become


    F(k) = nk-2/3k^3 ; V(k) = -nk^2/2+k^4/6




Fig. <ref> shows the evolution of potential energy as k varies for different G. Since external input G=0, the network dynamics is chaotic that results in the formation of potential wells that are both equally stable. The network activity will thus converge to any one of these wells (that can be interpreted as attractor states) depending upon the initial state or starting point. This supports the observation in <cit.> that a network with no input yields chaotic activity with incoherent and irregular trajectory for every trial. For nonzero G, the force (and potential) equation will be dependent on ζ since G ≃ cos(ωτ +2ζ). For different values of ζ, we solved for V (Eqn. <ref>) numerically and plotted the potential evolution as shown in Fig. <ref>. For ζ =π/4, the potential well is more attractive on the left end. This validates the fact that intrinsic fluctuations are suppressed in the presence of an input. For ζ =π /2, the left attractor becomes more stable. Changing ζ further shows that the potential well on the right end becomes more stable. This result confirms that input subspace alignment with respect to the initial chaotic state influences the overall stability and convergence capability of a recurrent network. 

The fact that stability corresponding to different attractor states (ζ =π /2, π) arises, qualifies our earlier hypothesis that input orientation with respect to the chaotic subspace alters the attractor state stability, corroborating the result of Fig. <ref> (f). Finally, we illustrate the effectiveness of input alignment on a complex motor pattern generation task with reliable generation of learnt handwritten patterns over multiple trials, even in presence of perturbations <cit.>. The detailed analysis and results are shown in the supplementary <cit.>. 

Note, we solved Eqn. <ref> by setting some initial and boundary value conditions on k and by iterating over different n until we reached a steady state solution. Changing these conditions will result in a completely new set of ζ values (different from those in Fig. <ref>). Nevertheless, we will observe a similar evolution of the potential well and change in attractor state stability as Fig. <ref>. Furthermore, the MFT calculations use ζ to denote a functional relationship between subspace alignment and input phase that eventually affects the attractor state stability. In the future, we will examine the real-time evaluation of ζ and its' impact on the analytical studies. Finally, the constraint under which we derive the potential energy functions and show the altering of attractor state is g = 1 + δ. We expect all our results to be valid for large g as well since Eqn. <ref> (that was simplified with Taylor expansion) still remains unchanged. 



§ CONCLUSION

Models of cortical networks often use diverse plasticity mechanisms for effective tuning of recurrent connections to suppress the intrinsic chaos (or fluctuations) <cit.>. We show that input alignment alone produces stable and repeatable trajectories, even, in presence of variable internal neuronal dynamics for dynamical computations. Combining input alignment with recurrent synaptic plasticity mechanism can further enable learning of stable correlated network activity at the output (or readout layer) that is resistant to external perturbation to a large extent. Furthermore, since input subspace alignment allows us to operate networks at low amplitude while maintaining a stable network activity, it provides an additional advantage of higher dimensionality. A network of higher dimensionality offers larger number of disassociated principal chaotic projections along which different inputs can be aligned <cit.>. Thus, for a classification task, wherein the network has to discriminate between 10 different inputs (of varying frequencies and underlying statistics), our notion of untangling with chaos-guided input structuring can, thus, serve as a foundation for building robust recurrent networks with improved inference ability. Further investigation is required to examine which orientations specifically improve the discrimination capability of the network and the impact of a given alignment on the stability of the readout dynamics around an output target. In summary, the analyses we present suggest that input alignment in the chaotic subspace has a large impact on the network dynamics and eventually determines the stability of an attractor state. In fact, we can control the network's convergence toward different stable attractor channels during its voyage in the neural state space by regulating the input orientation. This indicates that, besides synaptic strength variance <cit.>, a critical quantity that might be modified by modulatory and plasticity mechanisms controlling neural circuit dynamics is the input stimulus alignment. 







 §.§ Acknowledgments

P.P. and K.R. are supported in part by Center for Brain-inspired Computing (C-BRIC), an SRC and DoD sponsored center, the Semiconductor Research Corporation, the NSF, Intel Corporation and by the DoD Vannevar Bush Fellowship.





§ SUPPLEMENTARY MATERIAL



§ PCA FOR EXAMINING NETWORK ACTIVITY


To examine the structure of the recurrent network’s representations, we visualize and compare the neural trajectories in response to varying inputs using Principal Component Analysis (PCA) <cit.>. The network state at any given time instant can be described by a point in the N-dimensional space with coordinates corresponding to the firing rates of the N neuronal units. With time, the network activity traverses a trajectory in this N-dimensional space and we use PCA to outline the subspace in which this trajectory lies. To conduct PCA, we diagonalize the equal-time cross-correlation matrix of the firing rates of the N units as

    D_ij = <(r_i(t)-<r_i>)(r_j(t)-<r_j>)>

where the angle brackets, <>, denote time average and r(t) denotes the firing rate activity of the neuron. The eigenvalues of the matrix D (specifically, λ_a/∑_a=1^Nλ_a, where λ_a is the eigenvalue corresponding to principal component a) indicate the contribution of different Principal Components (PCs) toward the fluctuations/total variance in the spontaneous activity of the network. 
Fig. S1 shows the impact of varying input amplitude (I_0) on the spontaneous chaotic activity of the network. For I_0 = 0, the network is completely chaotic as is evident from the highly variable projections of the network activity onto different Principal Components (PCs) as shown in Fig. S1 (a, Left).
Generally, the leading 10-15% (depending upon the value of g) of the PCs account for ∼ 95% of the network's chaotic activity <cit.>. Visualizing the network activity in a 3D space composed of the dominant principal components (PC1, 2, 3) shows a random and irregular trajectory characteristic of chaos (Fig. S1 (a, Middle). In fact, plotting the trajectories (firing rate r(t) of the neuron as time evolves) of 5 recurrent units in the network (Fig. S1 (a, Right) shows diverging and incoherent activity across 10 different trials, also, representative of intrinsic chaos. In addition, the projections of the network activity onto components with smaller variances fluctuate more rapidly and irregularly (Fig. S1 (a, Left)). This further corroborates the fact that the leading PCs define a network's spontaneous chaotic activity. 

Driving the recurrent network with a sinusoidal input of high amplitude (Fig. S1 (b)) sensitizes the network toward the input, thereby, suppressing the intrinsic chaotic fluctuations. The PC projections of the network activity are relatively periodic. 

A noteworthy observation here is that the trajectories of the recurrent units (Fig. S1 (b, Right) become more stable and consistent across 10 different presentations of the input pattern with increasing amplitude. A readout layer appended to a recurrent network can be easily trained on these stable trajectories for a particular task. Thus, the input amplitude determines the network's encoding trajectories and in turn, its' inference ability. In fact, the chaotic intrinsic activity is completely suppressed for larger inputs. However, this is not preferred as input dominance drastically declines the discriminative ability of a network that can be justified by dimensionality measurements. The effective dimensionality of a reservoir is calculated as N_eff = ∑_a=1^N(λ_a^2)^-1 that provides a measure of the effective number of PCs describing a network's activity for a given input stimulus condition. Fig. S1 (c) illustrates how the effective dimensionality decreases with increasing input amplitude for different g values. It is, hence, critical that input drive be strong enough to influence network activity while not overriding the intrinsic chaotic dynamics to enable the network to operate at the edge of chaos. Note, higher g in Fig. S1 (c) yields a larger dimensionality due to richer chaotic activity. 

In our simulations in Fig. S1 (b), the input is shown for 50 ms starting at t =0. Thus, we observe that the trajectories of the recurrent units are chaotic until the input is turned on. Although the network returns to spontaneous chaotic fluctuations when the input is turned off (at t = 50 ms), we observe that the network trajectories are stable and non-chaotic that is in coherence with the previous findings from <cit.>. From the visualization of network activity in the dominant PC space, we see that the input-driven trajectory converges to a uniform shape becoming more circular (along PC1 and PC2 dimensions) with higher input amplitude (Fig. S1 (b)). This informs us that the orbit describing the network activity in the input-driven state consists of a circle in a two-dimensional subspace of the full N-dimensional hyperspace of the neuronal activities. Note, all simulations in supplementary are conducted with similar parameters mentioned in the manuscript, i.e., N=800, f= 10Hz, p_c =0.1.



§ MEAN FIELD DERIVATION WITH TAYLOR SERIES

First, let us solve for x_i^1 such that we can get an expression for the correlation function, C(τ) in Eqn. 3 of main manuscript. Noting that, x_i^1 is driven by Gaussian noise (as indicated by the MFT noise-interaction equation: dx_i^1/dt = -x_i^1 +η), we can assume their moments as <x_i^1(t)> = <x_i^1(t + τ)> = 0, <x_i^1(t) x_i^1(t)> = <x_i^1(t + τ) x_i^1(t + τ)> = Δ(0) and <x_i^1(t) x_i^1(t + τ)> =Δ(τ). x^1(t) (dropping index i as all neuronal variables have similar statistics) can then be written as
 
    x^1(t) = α z_1 + β z_3; 
    x^1(t+τ) = α z_2 + γ z_3

where z_1, z_2, z_3 are Gaussian random variables with 0 mean/unit variance and α = √(Δ(0) - |Δ(τ)|), β = sgn(Δ(τ))√(|Δ(τ)|),
γ= √(|Δ(τ)|). Now, writing x=x^0 + x^1, C is computed by integrating over z_1, z_2, z_3 as

    C(τ) =        1/N ∑_i=1^N <<ϕ (x_i^0(t) 
    + α z_1 + β z_3)>_z_1
           <ϕ (x_i^0(t+τ)) + α z_2 
     + γ z_3>_z_2>_z_3

where <f(z)>_z = ∫_- ∞^∞ dz f(z) exp(-z^2/2)/√(2π) for z = z_1, z_2, z_3. Now, x_i^0 (t) = A cos(2π ft + ζ), where A = I_0/√(1+(2π ft)^2) (solve dx_i^0/dt = -x_i^0 +I_0 cos(2 π ft + ζ) for x_i^0) and ζ incorporates the averaged temporal phase relationship between the individual neurons and the input induced by input subspace alignment. Replacing the value of x_i^0 in Eqn. <ref>, we get 

    C(τ) =        1/N ∑_i=1^N <<<ϕ (Acos(ζ)+ α z_1 + β z_3)>_z_1) 
           <ϕ(Acos(ωτ + ζ) + α z_2 + γ z_3>_z_2>_z_3)>_ζ

The above correlation function also satisfies Eqn. 4 of main manuscript. Note, ω = 2π f in Eqn. <ref>. 

Now we solve Eqn. <ref> using Taylor series approximation for tanh(gx) = gx-1/3g^3x^3 +2/15g^5x^5. We have

    ϕ(Acos(ζ) + α z_1 +β z_3)=       g(Acos(ζ) + α z_1 +β z_3)-1/3g^3(α z_1+β z_3)^3+2/15g^5(α z_1 + β z_3)^5 
    
    =       g(Acos(ζ) + α z_1 + β z_3) -1/3g^3(α^3 z_1^3 + 3 α^2 z_1^2 β z_3 + 3 α z_1β^2 z_3^2 + β^3 z_3^3) 
           +2/15 g^5(α^5 z_1^5 + 5 α^4 z_1^4 β z_3 + 10 α^3 z_1^3 β^2 z_3^2 +10α^2 z_1^2 β^3 z_3^3 
          +5 α z_1 β^4z_3^4+β^5z_3^5

Now using <z_1^2>_z_1 =1, <z_1^4>_z_1 =3, <z_1^6>_z_1 =15 and noting that averages over odd powers of z_1 are zero, we get
 
    <ϕ(Acos(ζ) + α z_1 +β z_3)>_z_1=       g(Acos(ζ) +β z_3)-1/3g^3(3 α^2 γ z_3 +γ^3 z_3^3)
          + 2/15g^5(15α^4 β z_3 + 10 α^2β^3 z_3^3 +β^5 z_3^5)

Similarly,
 
    <ϕ(Acos(ζ +ωτ) + α z_2 +γ z_3)>_z_2=       g(Acos(ζ +ωτ) +γ z_3)-1/3g^3(3 α^2 γ z_3 +γ^3 z_3^3)
          + 2/15g^5(15α^4 γ z_3 + 10 α^2γ^3 z_3^3 +γ^5 z_3^5)


Multiplying Eqn. <ref> with Eqn. <ref> upto the sixth order, we can simplify Eqn. <ref> as
 
    C =      <<g^2 A^2 cos(ζ) cos(ζ +ωτ) + g^2 βγz_2^2 -2 g^4α^2 βγz_3^2 -1/3 g^4(β^3 γ +βγ^3)z_3^2
          + 4g^6 α^4 βγz_3^2 + 4/3 g^6 α^2(β^3γ + βγ^3)z_3^4 +2/15 g^6(β^5 γ +βγ^5)z_3^6 
           +g^6α^4 βγz_3^2+1/3g^6α^2(β^3γ+βγ^3)z_3^4+1/9g^6β^3γ^3z_3^6>_z_3>_ζ

Now averaging Eqn. <ref> over ζ will still retain the ζ term unlike general MFT calculations where ζ gets washed out since there is no fixed relationship between the input and recurrent activity. Here, we use yet another approximation i.e. Cos(A). Cos(B) ≃ Cos(A+B) and finally get C as

    C =      1/2g^2 A^2 cos(2ζ +ωτ) +g^2 βγ - 2 g^4α^2 βγ - g^4 (β^3γ + βγ^3) +5g^6α^4βγ
          +5g^6α^2(β^3γ + βγ^3) +2g^6(β^5 γ +βγ^5)+5/3 g^6 β^3 γ^3

Using the definition of α, β, γ in Eqn. <ref> and denoting m=Δ(0), l=Δ(τ), we get

    α^2=m-|l|;βγ=l;β^3γ=βγ^2=|l|l; β^5γ=βγ^5=β^3γ^3=l^3

Substituting α, β, γ values from Eqn. <ref> in Eqn. <ref>, we get

    C =      1/2g^2 A^2 cos(2ζ +ωτ) +g^2 l-2g^4(m-|l|)l-2g^4|l|l+5g^6(m-|l|^2l +10g^6(m-|l|)|l|l+17/3g^6l^3 
    
    =       1/2g^2A^2cos(ωτ +2ζ) + l(g^2-2g^4m) + 2/3g^6l^3

Now, l satisfies d^2l/dt^2=l-C. By putting the value of C from Eqn. <ref> we get

    d^2l/dt^2=       1/2g^2A^2cos(ωτ +2ζ) + (1-g^2-2g^4m)l + 2/3g^6l^3

For deriving Eqn.5 from the main manuscript, we use l=kδ and define 1/2g^2A^2cos(ωτ +2ζ) = Gδ^3. Then, Eqn. <ref> becomes

    δ^3d^2k/dt^2=       -δ^3G+δ(-2δ+2(1+4δ)m)k-2/3δ^3k^3

Now, we introduce the parameter n, where m = δ +(n/2-4)δ^2 to further simplify Eqn. <ref> as

    d^2k/dt^2 = -G + n k -2/3k^3

Eqn. <ref> is the force equation (Eqn. 5 in the manuscript) from which we derived the potential energy. Note, the MFT approximate equations are derived with respect to a single input G driving the entire network. Multiple inputs (G_in where in=1,2...) and corresponding alignment of the inputs along different projections (as in Fig. 1 of main manuscript) will result in a potential well that can be roughly interpreted as a linear combination of V_in observed for each input, G_in. The linear combination will follow a similar evolution profile as shown in Fig. 2 of main manuscript. Thus, the change in intra-trajectory distance for varying alignment of the two inputs in the chaotic subspace, shown in Fig. 1 (f) of main manuscript, is justified from the given analysis. 




§ HANDWRITING GENERATION

To further elucidate the effectiveness of input alignment for complex pattern generation, we trained a recurrent network to generate the handwritten words “chaos” and “neuron” in response to two different inputs <cit.>. After obtaining the principal angle of the chaotic spontaneous activity, we aligned the input I_1 corresponding to “chaos” along ∠ PC1.PC2 using optimal input phase Θ. Then, we monitored the output activity for different orientation (i.e. 90^∘, 180^∘) of input I_2, corresponding to “neuron”, with respect to I_1 in the chaotic subspace. The two output units (representing the x and y axes) were trained using RLS to trace the original target locations (x(t), y(t)) of the handwritten patterns at each time instant. Fig. S2 (a) shows the handwritten patterns generated by the network across 10 test trials for the scenario when inputs are aligned at 90^∘ in the chaotic subspace. We observe similar robust patterns generated for 180^∘ phase as well. The notable feature of input alignment is that the chaotic trajectories become locally stable channels and function as dynamic attractor states. This can be visualized from the stable and synchronized trajectories observed for different neural units in Fig. 1 (b, Right) of main manuscript (and in Fig. S1). However, external perturbation can induce more chaos in the reservoir that will overwhelm the stable patterns of activity. 

To test the susceptibility of the dynamic attractor states formed with input structuring to external perturbation, we introduced random Gaussian noise onto a trained model along with the standard intrinsic chaos-aligned inputs during testing. The injection of noise alters the net external current received by the neuronal units (I = Σ_i[W_Input_i I + N_0 rand(i)], where N_0 is the noise amplitude, i denotes a neural unit in the reservoir and rand is the random Gaussian distribution). Fig. S2 (b) shows the mean squared error (calculated as the average Euclidean distance between the target (x, y) and the actual output produced at different time instants, averaged across 20 test trials) of the network for varying levels of noise. As N_0 increases, we observe a steady increase in the error value implying degradation in the prediction capability of the network. However, for moderate noise (with N_0 <0.01), the network exhibits high robustness with negligible degradation in prediction capability for both the words. Interestingly, for 90^∘ phase difference, “neuron” is more stable than “chaos” with increased reproducibility across different trials even with more noise (N_0 =0.2). In contrast, for 180^∘ phase, “chaos” is less sensitive to noise (Fig. S2 (b)). On the other hand, for 45^∘ phase alignment between I_1, I_2 in the chaotic subspace, we observe that the network is sensitive even toward slight perturbation (N_0 = 0.001). This implies that the attractor states formed, in this case, are very unstable. This further corroborates the fact that the extent of noise suppression and hence the attractor state stability varies based upon the input alignment. 




Fig. S3 shows the handwritten pattern generated in one test trial for different phase alignment between I_1, I_2, when I_1 is aligned along the principal angle defining the spontaneous chaotic activity of the network. It is noteworthy to mention that the neural trajectories of the recurrent units corresponding to both cases are stable. In fact, we observe in the 90^∘ case, the trajectories of neurons responding to I_1 that corresponds to output “chaos” become slightly divergent and incoherent beyond 1000ms. In contrast, the trajectories of units responding to the word “neuron” are more synergized and coherent throughout the 1500 ms time period of simulation. This indicates that the network activity for “neuron” converges to a more stable attractor state than “chaos”. As a result, we see that the network is more robust while reproducing “neuron” even in presence of external perturbation (N_0, noise amplitude is 0.2). In the 180^∘ phase difference case, we see exactly opposite stability phenomena with “chaos” converging to more stable attractor. 


10
url<#>1urlprefixURL 


dicarlo2012does
authorDiCarlo, J. J., authorZoccolan, D. &
  authorRust, N. C.
titleHow does the brain solve visual object recognition?
journalNeuron volume73,
  pages415–434 (year2012).

thorpe1996speed
authorThorpe, S., authorFize, D.,
  authorMarlot, C. et al.
titleSpeed of processing in the human visual system.
journalnature volume381,
  pages520–522 (year1996).

dicarlo2007untangling
authorDiCarlo, J. J. & authorCox, D. D.
titleUntangling invariant object recognition.
journalTrends in cognitive sciences
  volume11, pages333–341
  (year2007).

rajan2006eigenvalue
authorRajan, K. & authorAbbott, L.
titleEigenvalue spectra of random matrices for neural
  networks.
journalPhysical review letters
  volume97, pages188104
  (year2006).

kadmon2015transition
authorKadmon, J. & authorSompolinsky, H.
titleTransition to chaos in random neuronal networks.
journalPhysical Review X
  volume5, pages041030 (year2015).

stern2014dynamics
authorStern, M., authorSompolinsky, H. &
  authorAbbott, L.
titleDynamics of random neural networks with bistable
  units.
journalPhysical Review E
  volume90, pages062710
  (year2014).

van1996chaos
authorVan Vreeswijk, C., authorSompolinsky, H.
  et al.
titleChaos in neuronal networks with balanced excitatory
  and inhibitory activity.
journalScience volume274,
  pages1724–1726 (year1996).

van1998chaotic
authorvan Vreeswijk, C. & authorSompolinsky, H.
titleChaotic balanced state in a model of cortical
  circuits.
journalNeural computation
  volume10, pages1321–1371
  (year1998).

laje2013robust
authorLaje, R. & authorBuonomano, D. V.
titleRobust timing and motor patterns by taming chaos in
  recurrent neural networks.
journalNature neuroscience
  volume16, pages925–933
  (year2013).

sussillo2009generating
authorSussillo, D. & authorAbbott, L. F.
titleGenerating coherent patterns of activity from chaotic
  neural networks.
journalNeuron volume63,
  pages544–557 (year2009).

jaeger2004harnessing
authorJaeger, H. & authorHaas, H.
titleHarnessing nonlinearity: Predicting chaotic systems
  and saving energy in wireless communication.
journalscience volume304,
  pages78–80 (year2004).

S1ab
titleSee supplementary section 1 and fig. s1(a,b).

rajan2010inferring
authorRajan, K., authorAbbott, L. &
  authorSompolinsky, H.
titleInferring stimulus selectivity from the spatial
  structure of neural network dynamics.
In booktitleAdvances in Neural Information
  Processing Systems, pages1975–1983 (year2010).

ipsen1995angle
authorIpsen, I. C. & authorMeyer, C. D.
titleThe angle between complementary subspaces.
journalAmerican Mathematical Monthly
  pages904–911 (year1995).

S1c
titleSee supplementary section 1 and fig. s1(c).

rajan2010stimulus
authorRajan, K., authorAbbott, L. &
  authorSompolinsky, H.
titleStimulus-dependent suppression of chaos in recurrent
  neural networks.
journalPhysical Review E
  volume82, pages011903
  (year2010).

sompolinsky1988chaos
authorSompolinsky, H., authorCrisanti, A. &
  authorSommers, H.-J.
titleChaos in random neural networks.
journalPhysical Review Letters
  volume61, pages259 (year1988).

S2
titleSee supplementary section 2.

S3
titleSee supplementary section 3 and fig. s2, fig. s3.

panda2017learning
authorPanda, P. & authorRoy, K.
titleLearning to generate sequences with combination of
  hebbian and non-hebbian plasticity in recurrent spiking neural networks.
journalFrontiers in Neuroscience
  volume11, pages693 (year2017).

bertschinger2004real
authorBertschinger, N. & authorNatschläger, T.
titleReal-time computation at the edge of chaos in
  recurrent neural networks.
journalNeural computation
  volume16, pages1413–1436
  (year2004).

S5
titleThe handwriting patterns were adopted from laje et al. nature
  neuroscience doi:10.1038/nn.3405.



