{'fraction_non_alphanumeric': 0.03014411363917909, 'fraction_numerical': 0.007019862080356774, 'mean_word_length': 4.335536461775722, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Autonomous learning by simple dynamical systems with a discrete-time formulation

We present a discrete-time formulation for the autonomous learning conjecture. The main feature of this formulation is the possibility to apply the autonomous learning scheme to systems in which the errors with respect to target functions are not well-defined for all times. This restriction for the evaluation of functionality is a typical feature in systems that need a finite time interval to process a unit piece of information. We illustrate its application on an artificial neural network with feed-forward architecture for classification and a phase oscillator system with synchronization properties. The main characteristics of the discrete-time formulation are shown by constructing these systems with predefined functions.

The original formulation of the autonomous learning scheme works properly for systems where the cost functions are defined at all times during their evolution. However, many dynamical systems cannot satisfy this restriction since they need a finite time interval in order to process a piece of information and produce its response. As a result, the errors of such systems with respect to target functions are not defined during these processing intervals. Examples of these systems are feed-forward neural networks [2], spiking neural networks [3], gene regulatory systems with adaptive responses [4,5], signal transduction networks [6], etc.
In order to treat these kinds of systems we propose to define an iterative map for the evolution of their parameters. Between successive iterations, a dynamical system is allowed to evolve during a long enough transient time for its error function to be evaluated. With the error thus computed we update the parameters in the next iteration according to our proposed autonomous learning scheme. This approach can be seen as an adiabatic realization of the original learning scheme for continuous-time evolution.
To illustrate the application of this new formulation we consider two systems: a feedforward neural network and a Kuramoto system of phase oscillators. With the former, we have a classic example of a system that needs a time interval to process some input signal and produce its corresponding output. Since the system may classify several input patterns, the network does not only need a time to process each signal, it also needs to restart the process for each one of them. We show that this system can not only learn to classify a set of patterns but also to be robust against structural damages, namely, deletion of one of its nodes in the processing layer. In the case of the Kuramoto system, we repeat the problem of synchronization treated in our previous work [1], now using the new formalism. We then proceed to compare some aspects of the discrete and continuous-time approaches.
The work is organized as follows. In section two we write down the original continuoustime autonomous learning scheme, followed by the proposed discrete-time approach. In section three we describe the two systems where we apply the new scheme. In section four we present our numerical investigations of these systems. Finally, in the last section we discuss our results and present our conclusions.

II. AUTONOMOUS LEARNING THEORY
The continuous-time version of the autonomous learning scheme considers a simple dynamical system with N variables x = (x 1 , ..., x N ), and Q parameters w = (w 1 , ..., w Q ) with dynamics given by and an output function F (x) which defines the task the system is responsible of executing.
We further define a cost function or error ǫ between the output function and some target performance R 0 as Finally, to allow for a self-directed (or autonomous) minimization of the deviation ǫ, we extend the original system (1) by defining a dynamics for the parameters w as follows: In this expression δw(t) = w(t) − w(t − ∆) and δǫ(t) = ǫ(t) − ǫ(t − ∆) are temporal differences at time t and t − ∆, where ∆ is a time delay. The constant τ fixes the time scale of the evolution of w. In the last term, S plays the role of a noise intensity, and ξ(t) = (ξ 1 (t), ξ 2 (t), ..., ξ Q (t)) are independent random white noises with ξ(t) = 0 and Taken together, equations (1) and (3) define an extended dynamical system with combined variables x and w, evolving according to two different characteristic times. In effect, if the time scale of subsystem (1) is taken as unity and τ ≫ 1, the dynamics of subsystem (3) will be slower. Additionally, the time delay ∆ must satisfy τ ≫ ∆ ≫ 1.
Our conjecture is that, under an appropriate choice of τ , ∆ and S, this autonomous system will evolve along an orbit in the space of variables w which minimizes the system's deviation ǫ from the target performance. In [1] we show that this learning method works properly for systems of oscillators where different levels of synchronization must be reached.
Equation (3) has been designed in order to perform the weight evolution aiming to reduce the error ǫ. The interpretation of the first term on the right side of this equation is the following. If, as a consequence of the delayed feedback (memory), the system performance improves, i.e. δǫ < 0, with the weight correction satisfying δw i < 0, then the weight w i should next decrease. If instead δw i > 0, w i should increase. The opposite behavior is obtained in the case that the system performance decays with δǫ > 0. The second term on the right side of eq. (3) is a noise proportional to the error ǫ. Its function is to keep the dynamics from being trapped in local minima, vanishing when the error is zero.
Another interpretation of eq. (3) is that the weight evolution is controlled by a drift term (first term on the right) whose corrections are given by the memory (feedback) of the system. The second term is a stochastic exploration in parameter space proportional to the error ǫ. As a result, this dynamics can be seen as a competition between a drift term of intensity 1 τ and a stochastic term of intensity ǫS. As we will later show, a proper balance between these two terms is needed in order to obtain successful evolutions.

A. Discrete-time formulation
For the application of (1) and (3) it is necessary that the error ǫ(t) be defined for all t.
As we mentioned in the introduction, this restriction cannot be satisfied by systems which need a finite amount of time T to execute the task given by F (x) and, consequently, to yield a corresponding value for ǫ.
Our discrete-time autonomous learning approach for the optimization of such systems considers that during each time interval T the system's parameters w are fixed. After this transient, the system function F (x) assumes a value, allowing the error function ǫ to be evaluated and the parameters w to be accordingly updated. Taking each iteration step as comprising one complete processing interval T , we define an iterative map analogous to eq.
(3) as: Here, n is the iteration index, δw = w(n) − w(n − ∆ η ) and δǫ = ǫ(n) − ǫ(n − ∆ η ), with ∆ η ∈ N a delay given by a certain number of iterations. The other quantities are analogous to those of eq. (3). The constant K τ determines the characteristic time for the evolution of the parameters and is equivalent to 1/τ . In this way, the new formulation replaces (3) with an iterative map as the new dynamics for w, and allows the system to evolve according to (1) during an interval of time T between successive iterations.

III. DYNAMICAL MODELS
In this section we present the two models we set out to design through optimization with the discrete-time autonomous learning scheme. We note that the goal of these models in this work is only to serve as examples of application and we do not pretend to analyze their properties in detail nor to compare our procedure with other methods of optimization. We focus only on the autonomous learning procedure.
A. Neural network model The first example we consider is a classical feed-forward artificial neural network [2] able to classify bitmaps. This kind of system is prototypic for our interest. The neural network must process different signals with a fixed set of weights, requiring a finite time interval to compute the input information and retrieve its response. During this processing interval the error is not defined, making it difficult to implement the original scheme of autonomous learning.
Our model is defined as follows: a network G has N in nodes in the input layer, M nodes in the hidden layer and N out nodes in the output layer. A weight w c,c−1 ij is associated with a directed connection from node j in layer c − 1 to node i in layer c. The dynamics x c i of node i in layer c is where the activation function f is given by f (x) = tanh(x). For each node i in layer c, we include a threshold value in the activation function by adding a weight w c iθ from a threshold node θ to the node in question. The threshold node is always activated with θ = 1.
The neural network operates with a discrete-time dynamics. In the first iteration the input neurons read an input pattern ζ and get their values. In the second iteration the neurons in the hidden layer compute their state as a function of the states of the input neurons. Finally, in the third iteration the output nodes compute their states using the states of the neurons in the hidden layer. As a result, the output layer yields the response of the network after it processes the input pattern ζ.
A network may repeat this K times in order to compute the set of patterns ζ = ζ 1 , ..., ζ K .
We thus define the error of the network as In this expression y k i and x k i are the ideal and actual responses (respectively) of the output node i when the pattern ζ k is processed by the network.

Functionality
In order to construct functional networks, i.e., networks with small ǫ, we use our new discrete-time formalism of autonomous learning. We set the iterative map (4) as the evolution law for the weights of the neural network. In order to use this equation we sort all the weights w c,c−1 ij and w c iθ in one linear array w.

Robustness
We are also interested on retaining the functionality of a network when destructive mutations or damages alter the network structure. This property of structural robustness is a key feature of neural networks [2,8] and several biological systems [9]. Our aim is to employ the autonomous learning scheme as well to improve the robustness of a given system. The following ideas concerning robustness have already been applied to flow processing networks [10].
Consider a network G with error ǫ. If we delete one if its M nodes in the hidden layer, we get a new network G i with error ǫ i . In general, the damage introduced will worsen the functionality of the original network G, that is, we will have ǫ < ǫ i . We say that a damaged network is no longer functional if ǫ i > h, where h is thus defined as the maximum error below which a network is still considered functional. We can repeat this process of deletion for all nodes in the hidden layer to define the robustness ρ(G) of network G as the ratio between the number of damaged functional networks and the total number of possible damages M: Here, Θ is the Heaviside function with Θ(x) = 0 if x < 0 and Θ(x) = 1 if x ≥ 0.
Since our goal is the construction of functional and robust networks, we must aim for ǫ → 0 and ρ → 1. We propose the simultaneous optimization of these two quantities through the use of a biparametric autonomous learning scheme for the weights as follows: In this expression, we have two constants K ǫ and K ρ related to the drift term, and two constants S ǫ and S ρ related to the noise intensity. It is directly seen that this biparametric prescription for the evolution of w is essentially a superposition of (4) as applied individually to the optimization of ǫ and ρ. Note that the robustness must be calculated in each iteration, and therefore the M nodes are removed one by one in each iteration.

B. Kuramoto system
We take as a second example of a dynamical system the Kuramoto model studied in our previous article [1], now applying the discrete-time autonomous learning prescription for the evolution of the coupling weights.
The Kuramoto model [7] of coupled phase oscillators is described by equations where φ i is the phase and ω i is the natural frequency of oscillator i. The interactions are characterized by weights w ij . They are symmetric, i.e., w ij = w ji , and can be positive or negative.
The synchronization of the system is quantified by the Kuramoto order parameter Due to time fluctuations, we work with the mean order parameter R(t) defined as where T is the time interval we consider for its calculation. R(t) can vary between zero and one. In the case of full phase synchronization, R(t) → 1.
The aim of this example is to construct a system of oscillators able to autonomously learn to reach a target order parameter P . The error associated to this function is defined as The discrete-time autonomous learning scheme dictates that the system (9) evolve for a time T after each learning iteration. The iterative map for the weights is a variation of eq.
(4) for our system of oscillators: Note that we compute the weights w ij (with i, j = 1, ..., N) only for j > i, because of the symmetry of the interactions.
As in our previous work, we add a term to control the total interaction in the system.
The control parameter λ determines how strong the redistribution of the weights is, W being the ideal absolute value of v.
As a result of this dynamics the system of oscillators evolves to a mean order parameter R(t) = P by redistributing the weights and maintaining the total absolute value v(t) = W .

IV. NUMERICAL RESULTS
We present several numerical studies of the proposed systems.
We show examples of evolutions where the autonomous learning scheme is able to lead the system to the target states and we analyze particular aspects of it which arise in connection with each particular model.

A. Feed-forward neural network
The neural network we work with must learn to classify the vowels, i.e., there are K = 5 input patterns ζ. The letters are given as matrices of binary pixels, as shown in fig. 1.a.
Each pixel acts on an input neuron, with black squares indicating activation and white ones inactivation of the associated neuron.
The network has N in = 35 input neurons, one hidden layer with M = 15 nodes and N out = 5 output neurons. A schematic representation of the network is shown in fig. 1.b.

Functional networks
Our first experiment consists in the realization of a full evolution through the iterative map (4) to construct a functional network able to classify the letters. (a)

Convergence
As we have seen from the interpretations of eqs. 4 and 3, there is a competition between the drift term controlled by K τ and the stochastic term of exploration controlled by S. As a result, there exist certain combinations of these two values K τ and S where the learning The previous study considered only five input patterns for classification. Now, in order to get more robust results against the number of patterns to be classified, we consider 15 input patterns with the same network characteristics. This set of 15 patterns consists of the five vowels shown in fig. 1a and the ten numerical digits from 0 to 9. The number of output nodes is now 15. The mean error ǫ as a function of K τ and S is shown in fig. 3b. We observe that almost the same error surface is found as in the previous study, with the same optimum values for K τ and S.

Robust networks
We now consider the biparametric weight evolution given by eq. 8 that aims to minimize the error ǫ and maximize the robustness ρ. We set the values of K ǫ and S ǫ as the ones that guarantee the best convergence in the optimization of functionality, i.e., those found in the previous study. Performing a similar study to find the optimum parameters associated with the optimization of robustness, we found the minimum of 1 − ρ at (log K ρ = 0.75, log S ρ = −1.75). For the functionality threshold we take h = 0.0509.
An evolution for this case is shown in fig. 2.b. We observe that at the beginning of the learning process the error ǫ is high and, as a result, the network has zero robustness. When the error is reduced and close to the threshold h the learning of robustness is automatically turned on, owing to the fact that a damaged functional network has more chances to possess an error lower than the threshold as compared with a nonfunctional network with high error.
As the evolution progresses, the error is kept below h and the robustness increases until it reaches its optimum value. Thus, the resulting network is functional and robust.
In order to fix the threshold value h we proceed as in article [6]. We optimize an ensemble of 100 networks only by functionality (testing ensemble), computing the resulting final errors and those of the associated damaged networks. The corresponding histograms of the errors ǫ are shown in fig. 4.a (original ensemble) and 4.c (associated damaged ensemble) respectively.
We choose h as the value for which the functional ensemble has a mean robustness ρ = 0.50.
Hence, increasing the mean robustness from ρ = 0.50 to ρ = 1 will represent a 100% increase in robustness with respect to the testing ensemble.

B. Kuramoto system
This system presents an interesting characteristic concerning the error function. This function is evaluated by using the mean order parameter R(t) from eq. (11), that is, a mean value over a time interval T . In the previous formulation for continuous-time parameter dynamics the error can be evaluated at any time. However, we are then forced to make a prescription concerning the set of weights which are most responsible for this mean value.
In effect, during the interval T the weights are continuously changing and it is therefore not clear which set of assumed values during T are the effective ones in determining the error ǫ(t). The prescription we used then was that the error ǫ(t) be related to the weights at time we can always use a different prescription such as, for example, using the mean value of the weights over T . This problem has been previously considered in a different implementation of reinforcement learning for spiking neural networks [11].
A second problem related to this situation is that in the case of large enough T , the correlation between the weights and the error is missing. At the same time, we need a long enough period T in order to minimize the variations of R(t). This problem can be avoided by using our new formulation with discrete-time evolution for the parameters.
We take a full connected system with N = 10 phase oscillators with natural frequencies 1, 2, ..., 10). The integer delay is set to ∆ η = 1, and the target values to P = 0.6 and W = 0.3. As the initial conditions, we set w ij (0) = w ij (−∆ η ) = W and the initial phases φ i (0) uniformly distributed between 0 and 2π, which altogether results in the system's order parameter R having an approximate initial value of 0.3. Our aim is to increase the synchronization level to 100%. The system is numerically integrated using an Euler algorithm with time step dt = 0.01. Note that between iterations of the learning algorithm the phases φ preserve their values, therefore the initial conditions are not restarted as in our previous example. We use this protocol to accelerate the simulations and to avoid transients as well as cases of multistability.
1. An evolution  the initial and the final systems, respectively. The simulation is performed for a time interval longer than T . We observe the relative improvement in the behavior of the order parameter r(t) between the initial and final systems. The dashed black line shows the target value P prescribed for the learning evolution. We see that, for the final system, r(t) varies consistently around P .

Dependence on λ
Controlling the total absolute weight v(t) imposes a strong restriction on the learning process. Effectively, the corresponding correcting term implies that the difference W − v(n) is distributed in proportion to the strength of the connections. This way of redistributing the weights opposes their differentiation and, in general, resists the heterogeneities needed in order to find a solution. The stronger the correction by λ, the more difficult it is to reach small errors.   Figure 6.b we show the mean absolute weight v of these ensembles as a function of λ. We observe that, for large values of λ, the learning scheme cannot find good solutions and the mean value R is far from the target value P . However, the mean absolute weight v is near its target W with very small dispersion.
The opposite situation is found for small values of λ. There, the mean order parameter is much larger than W . As a compromise between these two tendencies, we may settle with λ = 0.01. In such a case, we find that 25% of the optimized networks have R > 0.5. For this sub-ensemble of networks, we have R = 0.57 and v = 0.306. Thus, the efficiency of the learning process to find acceptable solutions is about 0.25.

Dependence on time interval duration
We now study the dependence of the learning process on the time interval T . As done in the previous analysis, we optimize several ensembles of networks, this time fixing λ = 0.01 and varying T . We keep the previous values for the rest of the learning parameters. We observe that, for any value of T , we can find networks with order parameters close to the target P . However, for short periods of time T the fluctuations are strong and the weight restriction does not work properly. As a result, the total absolute weight v grows to values much larger than W . When we increase T we observe that the weight control operates correctly and all the solutions approach v = W . Additionally, it is interesting to note that the learning scheme works well for relatively short time windows of T = 50, considering that in our previous work [1] with the continuous-time version of autonomous learning we worked with a much longer time interval T = 200. This result indicates that the discrete-time formulation is more efficient than the continuous version in terms of convergence speed. This fact can be understood by considering that the weights values are fixed during the interval T in the new formulation and the error can be better estimated that way.

V. CONCLUSIONS AND DISCUSSIONS
In this work we presented a new formulation for the autonomous learning scheme by defining an iterative map for the evolution of the parameters of a dynamical system. The utility of this discrete-time formulation resides in including within the scope of application of the autonomous learning scheme those systems which need an intrinsic time interval of finite duration to process a unit amount of information and therefore cannot measure a cost function at all times during their dynamics.
The first system treated, a feed-forward neural network responsible for classifying several input patterns, is a typical example of a system subject to this restriction. We showed that our learning scheme works properly for this system, with the resulting networks able to classify the assigned patterns. Furthermore, we showed that we can implement the discretetime scheme in biparametric form by setting a double feed-back signal in the evolution prescription for the weights, allowing us to optimize the networks with respect to the error ǫ and the robustness ρ simultaneously. Our results show that the final systems can in this manner improve their robustness 66% with respect to a testing ensemble.
It is important to mention that from the point of view of machine learning [12], the autonomous learning scheme can be classified as a type of reinforcement learning. These methods are characterized by their slow convergence, mainly due to the stochastic exploration of parameter space. In our case, this exploration is carried out by the multiplicative noise. As a result, our method is found to converge slower than the classical back-propagation algorithm, which constitutes a form of supervised learning by gradient descent.