abstract: We redefine the transition function of elementary cellular automata (ECA) in terms of discrete operators. The operator representation provides a clear hint about the way systems behave both at the local and the global scale. We show that mirror and complementary symmetric rules are connected to each other via simple operator transformations. It is possible to decouple the representation into two pairs of operators which are used to construct a periodic table of ECA that maps all unique rules in such a way that rules having similar behavior are clustered together. Finally, the operator representation is used to implement a generalized logistic extension to ECA. Here a single tuning parameter scales the pace with which operators iterate the rules. We show that, as this parameter is tuned, many rules of ECA undergo multiple phase transitions between periodic, locally chaotic, chaotic and complex (Class 4) behavior.
address: Centre de Physique Théorique (CPT), Turing Center for Living Systems, Aix Marseille Université, 13009 Marseille, France; UNAM-Institute of Materials Science and Nanotechnology, Bilkent University, Ankara 06800, Turkey; Department of Electrical and Electronics Engineering, Bilkent University, 06800, Ankara, Turkey; Department of Aeronautical Engineering, University of Turkish Aeronautical Association, 06790, Ankara, Turkey; Department of Physics, Bilkent University, Ankara 06800, Turkey; Interdisciplinary Graduate Program in Neuroscience, Bilkent University, 06800, Ankara, Turkey
author: M. Ibrahimi; A. Güçlü; N. Jahangirov; M. Yaman; O. Gülseren; S. Jahangirov
title: Operator representation and logistic extension of elementary cellular automata

Emergence, a semantic gap between behavior and interactions, is a hallmark of dynamical systems. Examples of emergent behavior include: exchanging deals between thousands of agents/companies making up the whole stock market; interactions between alternately expressing genes generating juxtaposed biological forms; series of firings between peculiarly interconnected neurons breeding functional activities in the brain. Given the particularities of such systems (multiple levels of hierarchies, events and processes operating at broadly different space/time scales), complexity science has adopted cellular automata (CA) as much simpler computational models to specifically target the semantic gap between individual and global degrees . These agent based models operate in fully discrete domains and are known to generate large scale types of behavior only through local interactions (rules) .

While aiming to acquire a generic understanding applicable to all dynamical systems, a main hypothesis has gathered several attempts to bridge the spatio-temporal patterns observed in CA (phenotype) with their rule space (genotype). The leading apprehension is Wolfram's classification which postulates that the asymptotic behavior of a dynamical system lies in one of these four classes: homogeneous, periodic, aperiodic and complex behavior . He introduced elementary cellular automata (ECA) as a paradigmatic simple set of rules which comprise all these types.

Several studies have attempted to understand how distinct groups of rules act to generate similar types of asymptotic behavior, eventually making up the four classes. Langton's method of labelling a parameter out of a unit interval to a certain rule is a simple, yet often efficient approach for predicting the class . Also, further approaches have introduced entropies, mean field descriptions or network analyses that help to understand the relation between patterns and their respective rules . However, these analyses are either too generic to hold for, or too specific to apply to larger families of CA. Consequently, the quest for generalizing a method to any dynamical lattice system remains challenging and this motivates the need to approach CA rules in the light of a different perspective.

Using the ECA set as an example, we suggest a fundamental approach that redefines the transition function based on a simple intuition gained by visual inspection of the system scale dynamics. This approach brings the microscopic information closer to the large scale dynamics and thus helps understand the properties of a system based on its "first principles".

Having distilled the interactions into iterative operations, we employ these operations and the symmetries of the system to rewrite the transition function in a more intuitive notation. Our approach provides a framework that links the similarities and differences observed at the phenotype level to an operator based translation of the rule space. Furthermore, this framework enables us to implement a generalized logistic extension to ECA where a single parameter scales the pace with which operators iterate the system. As a result, the binary state space of ECA is expanded into a Cantor set and in turn we get a chance to observe transitions between classes . In particular, we reveal several complex (Class 4) instances that are not reachable in the standard ECA. More interestingly, the behavioral difference between some rules sharing similar genetic code is diminished upon the logistic extension.

ECA are time dependent one dimensional infinite strings of sites $S^t = \{s_{n} ^t\}|_{n = -\infty} ^\infty$ of a binary state space $s_i \in \{ 0 , 1 \}$. In ECA a Rule defines how the value of a certain site is iterated $s^{t+1} _i = f_S s^{t} _i$ based on its current value and the values of its nearest neighbors, through a transition function $f_S (s^{t} _{i-1}, s^{t} _i, s^{t} _{i+1})$. Given the binary state space, there are eight possible configurations of a three site neighborhood, resulting in $2^8 = 256$ possible mappings, i.e rules. Mapping of Rule 30 is shown as an example in Fig. 1(a). The name "30" of this rule comes from the binary to decimal transformation of the string 00011110 obtained from the particular mapping of the eight configurations listed in the order shown in Fig. 1(a). ECA possesses two important symmetries: complementary and mirror. Here complementary means flipping the state in every site of the array. Hence, if Rule A and Rule B are complementary (mirror) symmetric, then running Rule A with a certain initial string will give the complementary (mirror) image of running Rule B with the complementary (mirror) version of that string. When these symmetries are taken into account, the number of unique rules reduces to 88 (and not 64 since mirror and/or complementary symmetries of certain rules are equivalent to themselves).

Simple observations on ECA runs reflect visual structures of uniform, stable, oscillatory or irregular patterns. These structures are prone to a mixture of three types of fundamental iterations \[$s^{t} _i\rightarrow s^{t+1} _i$\], namely: decay \[$0|1\rightarrow0$\], stability \[$0(1)\rightarrow0(1)$\], and growth \[$0|1\rightarrow1$\]. Note that this approach becomes evident in a numerical representation of the state space, and it is the key in translating the rules into discrete operators. We first use the mirror symmetry of these systems to regroup the eight possible configurations into symmetric and asymmetric sets, as seen in Fig. 1(a). Then, within each set, we group complementary configurations together, leading to four groups (denoted by Roman numerals in four different colors). Each group has central cells with values 0 and 1 that are mapped in four different ways: \[$0\rightarrow0$, $1\rightarrow0$\], \[$0\rightarrow0$, $1\rightarrow1$\], \[$0\rightarrow1$, $1\rightarrow0$\], and \[$0\rightarrow1$, $1\rightarrow1$\]. We call these double mappings operators, and conveniently name them as Decay (D), Stability (S), Oscillation (O), and Growth (G), respectively. Note that the oscillation operator is a compound of decay and growth iterations. Four groups and four possible operators cover all 256 rules ($4^4 = 2^8$). The operator representation of Rule 30 becomes DSOG. Symmetric counterparts of rules are easily constructed in operator representation. As shown in Fig. 1(b), to get the mirror symmetry of a rule, one needs to switch the operators in the group III and group IV. To get the complementary of a rule, one needs to replace all D operations (if any) with G and vice versa. Symmetries of the Rule 30 (DSOG) found by these transformations are presented as an example in Fig. 1(c).

Symmetric (I and II) and asymmetric (III and IV) sets of operators are decoupled from each other with respect to both mirror and complementary transformations. Hence, it is instructive to arrange ECA in a "periodic table" by placing possible symmetric sets as abscissa and asymmetric sets as ordinate. However, using all 16 pairs of operations in both axes leads to many repetitions of rules that are identical under mirror and complementary transformations. This can be avoided by realizing that, for example, a symmetric set "DO" becomes "GO" under complementary transformation while remaining the same under mirror transformation. Omitting one of these pairs erases a whole column of repetitions. Continuing in this fashion one can reach at a 10$\times$`<!-- -->`{=html}10 table that has all 88 unique Rules with only 12 repetitions. While constructing this table, one needs to decide which repeating columns to erase and how to arrange the rows and the columns that are left at the end. The table that we have constructed, after evaluating numerous options based on mathematical and aesthetic criteria, is presented in Fig. 2. The 12 repetitions that appear at the corners of the table are removed for clarity. Note that, every adjacent row and column share at least one common operator which means that every adjacent rule on the Table share at least three common operators.

The Periodic Table presented in Fig. 2 offers a systematic "bird's eye" view of all 88 unique rules of ECA. Rules dominated by similar simple patterns (homogeneous, vertical lines, diagonal lines, horizontal stripes) tend to appear together. The rules that show rich behavior populate the "fertile crescent" along the diagonal where simple rules with contradicting patterns are expected to overlap. Among these rich rules, the ones that have common features are also brought together. Rule pairs 18, 146 and 122, 126 are striking examples of this. Despite the chaotic nature of these rules, starting a run with one of them and switching to the other rule results in the same pattern that is produced without the switching. This is because, Rule 18 (122) and Rule 146 (126) share the same mapping, except for the configuration 111 (010) which is mapped to 0 in the former and 1 in the latter. This 111 (010) configuration is "washed out" in a few steps and is never visited again. This effect is also present if one starts with the Rule 26 and continues with the Rule 154 but not the other way around.

The Periodic Table of ECA also resonates with the findings of Li and Packard in their classic study on the structure of the ECA rule space. They found two clusters of chaotic rules (in this context it includes the complex rules 54 and 110). Chaotic A includes Rules 18, 22, 30, 54, 146, and 150 while Chaotic B has Rules 60, 90, 106, 110, 122, and 126. As seen in Fig 2, they appear as clusters at the bottom left and top right of the "fertile crescent", respectively. The authors found Rule 45 to be separated from the clusters but in our Table we find it connected to the cluster B. Furthermore, clusters A and B are connected over a bridge of locally chaotic Rule 26 in the Table. There are no other chaotic rules in the row and the column of the Rule 105, which was also found to be isolated by Li and Packard, but it is connected to the cluster B over a bridge of locally chaotic Rule 73.

The operator representation can further illuminate the studies on the computational irreducibility of ECA. In particular, it is interesting to examine the rules that are detached from the coarse-graining network investigated by Israeli and Goldenfeld . They have shown that, Rule 105 can be course grained by the Rule 150. In the operator representation, these rules appear as OOSS and SSOO, respectively. Furthermore, both DGDG (Rule 60) and DDGG (Rule 90) can be coarse-grained by themselves. Finally, the authors were unable to coarse-grain four unique rules: 30, 45, 106 and 154. In the operator representation, they happen to be DSOG, OGDS, OSGD, and SDOG. These make up four unique rules that involve all four operators while avoiding two complementary symmetric operators (D and G) in the same mirror symmetric set. In other words, the rules that were found to be irreducible are the ones that appear the most asymmetric in the operator representation. We believe that these mere observations can guide further studies in this subject.

Recently, we have introduced the logistic extension of two outer-totalistic CA: Game of Life and Rule 90. This extension is achieved via introduction of a parameter, $\lambda$, that tunes the dynamics of CA. $\lambda = 1$ corresponds to the original binary version of the studied systems. As $\lambda$ is tuned below 1, the binary state space extends into a Cantor set and the systems expand their complexity through series of deterministic transitions . In particular, the Rule 90 which is aperiodic at $\lambda = 1$ shows complex (or Class 4) behavior at $\lambda \sim 0.6$. The operator representation presented here enables us to go beyond the outer-totalistic rules and generalize the logistic extension to all ECA. We first define four regions of operation for each group (I, II, III and IV) as shown in Fig. 3. The coordinates of a configuration $[L, C, R]$ (denoting left, center and right sites, respectively) defined as the sums $x \equiv L+R+\frac{1}{2} \pmod{2}$ and $y \equiv L+C+\frac{1}{2} \pmod{2}$ determine in which operation region it falls. As shown in Fig. 3, the eight possible binary configurations appear at the centers of the regions that correspond to their group definitions shown in Fig 1(a). Hence, the configuration $[L, C, R]$ determines the operation region which in turn determines the corresponding operator based on the rule at hand. Depending on the operator, the value of a site is updated according to the following formulae:

$$\begin{split}
\text{Decay} ~ \Rightarrow ~ & s^{t+1} = (1 - \lambda) s^t \\
\text{Stability} ~ \Rightarrow ~ & s^{t+1} = s^t \\
\text{Oscillation} ~ \Rightarrow ~ & s^{t+1} =
    \begin{cases}
      (1 - \lambda) s^t + \lambda, & \text{if}\ s^t \leq \frac{1}{2} \\
      \\
      (1 - \lambda) s^t, & \text{if}\ s^t > \frac{1}{2}
    \end{cases}  \\
\text{Growth} ~ \Rightarrow ~ & s^{t+1} = (1 - \lambda) s^t + \lambda \\
\end{split}$$

where $s^t$ and $s^{t+1}$ are the values of the central site at the current and the next time step, respectively. These equations, consistent with the operator notation, make up the new form of the transition function. Note that this generalization is consistent with the special case of the Rule 90 that we have reported earlier .

Significant changes in dynamics can occur when $x$ or $y$ passes over from one region to another. This happens when the sum $L+C$ or $L+R$ is equal to the critical thresholds 0.5 or 1.5. The values that $L$, $C$, and $R$ can take is dictated by the $\lambda$-dependent Cantor set. Hence, one can expect these changes at the values of $\lambda$ that mark the equality of binary sums to the critical thresholds. As $\lambda$ is tuned below 1, the first time such a transition occurs is when $2 \lambda = 1.5$. After this point, some of the rules start behaving differently than their original version.

Rules that exhibit chaotic, locally chaotic or complex behavior pass through multiple phase transitions while going between these regimes. As seen in Fig. 4, chaotic Rule 18 becomes complex at $\lambda = 0.73$ mimicking (but not exactly copying) the complex patterns seen in one of its neighbors, Rule 54. Another locally chaotic rule close by, Rule 82, also mimics the Rule 54 behavior at $\lambda = 0.74$.

Logistic extension breaks the symmetry between mirror rules because of the left-right asymmetry in the sum $L+C$. This is clear in the distinct behavior of Rule 26 (the mirror symmetry of Rule 82) which has a mixture of chaotic and locally chaotic behavior at $\lambda = 0.74$. However, complementary rules behave in the same way under the logistic extension. For example, the behavior of complementary Rules 90 and 165 are the same at $\lambda = 0.6$ . Rules that originally have complex behavior may remain complex while having noticeable changes in their dynamics, for example Rule 54 at $\lambda = 0.74$. They also can become locally chaotic like Rule 110 at $\lambda = 0.74$ or become chaotic like Rule 124 (not shown in Fig. 4) which resembles its neighboring chaotic Rule 60 at $\lambda = 0.72$. Rules that are chaotic or locally chaotic can behave in a complex fashion as exemplified by Rule 86 (mirror symmetry of the Rule 30) at $\lambda = 0.68$ and Rule 154 at $\lambda = 0.68$, respectively.

Some of the Rules that are originally periodic can acquire aperiodic behavior. For example, Rule 38 becomes locally chaotic at $\lambda \sim 0.69$ and chaotic at $\lambda \sim 0.61$. Periodic rules can also become complex, for example Rule 37 and Rule 46 at $\lambda = 0.72$, as shown in Fig. 4. Rules that gain aperiodic behavior upon the logistic extension are highlighted by the green squares in Fig. 2. Note that, these rules are adjacent to the rules that are originally aperiodic.

In summary, to understand the disjunctive and connective (diverse and unifying) nature of ECA rules, we redefine the transition function by introducing an operator-based notation. This allows one to organize the rules in a periodic table where underlying connections between their macroscopic behaviors and genetic codes emerge. Furthermore, we introduce a tuning parameter which controls the rate of rule iterations. This parameter extends the range of behavior that ECA can offer while generating inter-class transitions and disclosing inert behaviors of periodic rules. We believe that logistic extension to the operator-based representation may be useful to explore hidden features in other complex systems, such as discrete lattice models and boolean genetic networks .

S. J. acknowledges support from the Turkish Academy of Sciences - Outstanding Young Scientists Award Program (TÜBA-GEBİP).


abstract: Dedicated machines designed for specific computational algorithms can outperform conventional computers by several orders of magnitude. In this note we describe *Ianus*, a new generation FPGA based machine and its basic features: hardware integration and wide reprogrammability. Our goal is to build a machine that can fully exploit the performance potential of new generation FPGA devices. We also plan a software platform which simplifies its programming, in order to extend its intended range of application to a wide class of interesting and computationally demanding problems. The decision to develop a dedicated processor is a complex one, involving careful assessment of its performance lead, during its expected lifetime, over traditional computers, taking into account their performance increase, as predicted by Moore's law. We discuss this point in detail.
address: Physics Department, Università di Ferrara, I-43100 Ferrara (Italy); INFN, Sezione di Ferrara, I-43100 Ferrara (Italy); Instituto de Biocomputación y Fı́sica de Sistemas Complejos (BIFI), Zaragoza; Departamento de Fı́sica Teórica, Facultad de Ciencias,
Universidad de Zaragoza, 50009 Zaragoza, Spain; Departamento de Ingenieria Electrónica y Comunicaciones,
Universidad de Zaragoza, CPS, Maria de Luna, 1, 50018 Zaragoza, Spain; Departamento de Fı́sica Teórica I, Facultad de CC. Fı́sicas,
Universidad Complutense de Madrid, 28040 Madrid, SPAIN; Dipartimento di Fisica, Università di Roma "La Sapienza"
P. A. Moro 2, 00185 Roma, Italy; INFN Sezione di Roma; Departamento de Fı́sica, Facultad de Ciencias,
Universidad de Extremadura, 06071 Badajoz, SPAIN,; Servicio de Instrumentación Cientı́fica, Universidad de Zaragoza.
author: F. Belletti; I. Campos; A. Cruz; L. A. Fernández; S. Jiménez; A. Maiorano; F. Mantovani; E. Marinari; V. Martı́n-Mayor; D. Navarro; A. Muñoz Sudupe; S. Pérez Gaviro; G. Poli; J. J. Ruiz-Lorenzo; F. Schifano; D. Sciretti; A. Tarancón; P. Téllez; R. Tripiccione; J. L. Velasco
title: *Ianus*: an Adaptive FPGA Computer

# Introduction

Dedicated computers have had a long history vis-a-vis several physics problems demanding huge computing resources, where conventional computers have not allowed to reach conclusive results. A non exhaustive list of these problems includes Monte Carlo simulations of Lattice QCD , numerical studies of large self-gravitating systems and high-resolution numerical solutions of the Navier-Stokes equations in the turbulent regime .

Statistical mechanics is another area, relevant for our plans, where dedicated computers have also been heavily used , starting from the pioneering efforts dating as far back as the late seventies  for the simulation of the Ising model. Several dedicated machines have been developed more recently to study spin-glass systems. A dedicated machine, *RTN*, built in Zaragoza in 1991 , was designed to perform computationally intensive calculations that make limited use of floating point arithmetics, such as spin glasses and lattice Gauge-Higgs model simulations. RTN was based on transputer processors: it had 8 identical boards, each with 8 transputers, one connection chip, and one controller board. RTN was at the time a very effective and reliable computational device. *SUE* (Spin Update Engine) has been the second generation spin-glass machine . SUE was completed in the year $2000$, and it has been used to simulate three dimensional Ising spin glasses. It consists of twelve identical boards: each board simulates at the same time $8$ different systems. The update speed of the whole machine is $217$ ps/spin (when running at a $48$ MHz clock frequency). Field Programmable Gate Arrays (FPGA) made it possible to design a low cost, high performance, reliable dedicated machine. The FPGA-based architecture allows to reprogram the structure of the hardware connections of the computer: in SUE this feature was used to perform simple changes, like studying systems of different sizes, or modifying the updating dynamical scheme, or to change some details of the Hamiltonian of the system.

SUE was very different from RTN: instead of "usual" processors like transputers (but remember that even transputers were not usual at all in their extreme optimization toward easy inter-communication) it adopted FPGA's (of a generation which, looked at from today, seems prehistoric): these devices give the possibility of "programming" the hardware and of configuring the processor at best. In short, they contain logical gates that can be activated or deactivated or connected as needed, allowing in this way to select the desired functionalities.

In this paper we describe yet another generation spin-glass computing engine, called *Ianus* (after the name of the two-faced ancient Roman God of doors and *gates*), whose development is well underway. The new project has basically two goals: i) developing an extremely high-performance spin-glass simulation engine, able to update (on average) one spin in less than $1$ ps and ii) developing a software infrastructure that drastically reduces the effort to customize an array of FPGA's to a specific computational algorithm.

This paper is structured as follows: in section $2$ we discuss the advantages and disadvantages of building dedicated computers instead of buying commercial clusters, and in section $3$ we describe as a practical example our computational goals for spin-glass simulations and the performance of some non-dedicated computers when handling it. Section $4$ describes the overall architecture of the new machine and section $5$ discusses the hardware main issues. Section $6$ presents performance estimations, and the paper is ended by some concluding remarks that also cover longer term plans to make *Ianus* more general purpose.

# Advantages and disadvantages of dedicated computing

The decision to invest time and money in developing a new dedicated computer has to follow an accurate balancing between real benefits and development burden; also the possibility of seeing the project *aging* when compared to actual scientific progress and technological improvements has to be taken into account.

Optimized computers provide much larger computing power when used in solving problems they are designed for. This can show up either as a much better "price to performance" ratio, so that the computing power available within the limits of a given budget is greatly increased, or as a much higher sheer performance, technically not achievable at a given point in time on the given problem with commercial computers. This is not the only way dedicated computers shorten the time needed to produce scientific results: usually the development of computer applications for scientific research is centered in squeezing performance out of very generic hardware. Code optimization is always a hard and time-eating task and often the gain is not comparable to the effort. A dedicated machine is built *around* a specific computational task, and therefore the final user always accesses its maximum performance.

All computational problems for which dedicated machines have been proposed have a very large degree of regularity, that typically translates into heavy use of *unusual* mathematical sequences and into extreme levels of parallelization. Both features are key factors for performance:

-   Traditional computers are optimized for execution of short linear sequences of code performing irregular memory accesses, and integer arithmetic or logic manipulations (adds/subtract) of long data words ($32$ or $64$ bits). Recent designs also focus on floating point arithmetics. Every time the mix of required operations is at variance with the above list, there is room for dedicated processing. For instance, the Grape processor is centered around an hardware block able to compute on the fly the inverse of a square-root. In Lattice QCD virtually all processing amounts to manipulation of complex matrices so that hardware pipelines computing $a \times b + c$ (with $a, b, c$ complex numbers) are extremely effective. In our case, as discussed in details later on, we leverage on the fact that our typical operations are logical operations on a small number of bits and that the data-base to be processed is small enough that we store it within the processor, fully removing the bottle-neck of memory access.

-   Traditional computers are not usually designed to be effective for massive, regular and tightly coupled parallelism. Rather, typical design goals in commercial parallel systems are extreme flexibility in possible communication patterns, at the price of interconnection bandwidth and latency. Many computationally intensive applications on the other hand have intrinsically high levels of parallelism, so each processing node handles a very small subset of the physical system. This brings inevitably to a large ratio of information exchanged among nodes over performed arithmetic/logic operation. Parallel efficiency requires that the time $T_p$ needed by a processor to handle (e.g., on one iteration) its system subset is roughly equal to the time $T_c$ needed to exchange needed data with other nodes. We will see that this requirement is badly violated in traditional computer systems, while dedicated structures may leverage on regularity of communication patterns to achieve the goal of $T_p
    \simeq T_c$ with limited resources.

All this has costs.

-   Large investment in time: a new concept dedicated machine always requires a long development time, due to designing, building and testing it.

-   Large investment in man-power: specific high-level knowledge in electronics, digital design and topics in the target machine's application fields are needed. Working groups made of engineers, physicists and biologists have to be formed and coordinated. In Universities, the investment in the training of collaborating students has to be taken into account.

The relatively long time frame associated to development carries two additional risks.

-   Possible loss of relevance of the optimized algorithms. A lot of care has to be taken when judging what shall be still interesting to the scientific community in the following years, to avoid that the specific problems on which the new machine is unbeaten becomes obsolete before the machine does. We may say that it is the machine that has to make the problem grow old and not the opposite.

-   Loss of competitiveness in time with respect to commercial computers. Computer performance has been growing steadily according to Moore's law (performance doubles every approximately 18 months), while the performance of a dedicated machine remains constant, till a new generation is developed. Fig. sketches the situation: one must be sure that the performance gain associated to the dedicate system, *when* it comes into operation is large enough to offset power improvements of traditional processors working in parallel with a reasonable parallel factor, and provide a window of opportunity of at least a few years.

In the next sections we analyze these problems with a specific regard to our application.

# Monte Carlo simulations of spin-glass systems

The first area of application for *Ianus* is the accurate and detailed study of spin-glass systems. Later on we will describe other contexts in which we expect *Ianus* to become a key player.

We consider here the the Edwards-Anderson spin glass model  in three dimensions. We define as usual a collection of variables $\sigma_i\ ,\ i=1,\dots,N=L^3$ corresponding to sites of a cubic lattice of size $L$; each variable takes values $\{-1,+1\}$. A collection of static links $J_{ij}\in \{-1,+1\}\ ,\
i,j=1,\dots,N$ between first neighbor sites determines the properties of the system, and we are interested in randomly (quenched) chosen ones. When studying equilibrium properties of the system, averages on a large number of realizations of the $\{J\}$ ensemble are appropriate, while an analysis of the system away from equilibrium requires very long time histories over a smaller set of $\{J\}$. In the former case, there is an additional opportunity for trivial parallelization, as simulations are performed in parallel over different sets of $\{J\}$, that possibly use the same random numbers. This option is much less effective in the latter case, whose physical interest is growing with time, and for which we plan to optimize our new processor.

We discuss here two different algorithms, namely the Demon   and the Heat Bath Algorithm  . The main advantage of the former algorithm is that random number are not needed, so it was widely applied in earlier dedicated machines. For the latter algorithm, on the other hand, performance depends strongly on the efficiency of (multiple) random number generators.

The energy of the system is defined by $$\label{eq:USG}
U = -\sum_{<ij>}\sigma_i J_{ij} \sigma_j \;,$$ where $<ij>$ indicates that sums have to be performed over all pairs of neighboring sites in the physical lattice. In the Heat Bath algorithm spins are updated by extracting their values with a probability given by: $$\label{eq:PHB}
P(\sigma_i = 1) = \frac{ \e^{\sum_{<ij>} J_{ij} \sigma_j /kT } }
{ \e^{\sum_{<ij>} J_{ij} \sigma_j /kT}
+ \e^{-\sum_{<ij>} J_{ij} \sigma_j/kT }}\;.$$ Updates may be done sequentially (one at a time, eventually sweeping the whole lattice) or in parallel, being careful to respect balance. The highest level of achievable parallelism is obtained when all sites are divided into two groups in a checkerboard configurations and all elements of each group are updated at the same time (use of checkerboard structure is needed to ensure that detailed balance is respected during the simulation).

In the Demon algorithm, the system is coupled to a set of energy reservoirs ("demons") $e_k\ ,\
k=1,\dots,K$. The energy of this enlarged system is given by $$U = -\sum_{<ij>} \sigma_i J_{ij} \sigma_j + \sum_ke_k\;,$$ where all the demons may be initially set to zero. The demons cannot lower their energy below zero and also an upper limit may be assigned. Consider the case $K=1$ with a unique reservoir coupled to the system. Spins are updated sequentially depending on the actual values of the reservoir: if inverting the value of a spin is energetically convenient, the demon receives the energy variation and the spin is flipped, otherwise the spin flips only if the reservoir may supply the needed energy.

Both algorithms can be coded in traditional processors with reasonable efficiency.

For instance, the Heat-Bath procedure updates a single spin every 10 nanoseconds on a Pentium IV processor at 3.2 GHz by means of a reasonably programmed multi-spin coding updating routine, in which one random number is used for each try. If multiple spins (belonging to independent ensembles) use the same random number performance increase by a factor of roughly $3$.

Both algorithms can be parallelized very easily. Typically the physical lattice is divided in homogeneous region and computations in each of them are carried out by one computational node; only boundaries of each partition have to be transported between nodes after each step of the algorithm. We have parallelized the Heat-Bath algorithm on a set of 4 Pentium IV nodes at 3.2 GHz with Gigabit Ethernet obtaining a parallel efficiency of 42%, when considering a lattice of global size $64^3$. The small amount of information to be transmitted per each complete Monte Carlo lattice update makes latency overhead very large for each communication. The actual spin update time (within each node of the cluster) to consider when comparing to our new dedicated machine is then about 24 nanoseconds.

A properly coded routine for the demon algorithm updates a spin in less than 4 nanoseconds. The parallel efficiency in this case is even lower, about 25%. Sustained parallel update time in this case is therefore of the order of $15$ nanoseconds in this case.

Very recently, new computer options have become available, in terms of very massively parallel systems with nearest neighbor connections (Blue Gene/L ) and recently announced processors with extremely powerful on-board processing resources (like the Clearspeed CSX600 processor or the IBM/Sony/Toshiba Cell processor ).

In order to estimate the performance of spin-updates on these new machines, we may build a very simple model, in which we define the computational load as $\lambda l^3$ ($l$ is the linear size of the lattice handled by each processor), and the information-exchange between neighboring nodes $6 \iota l^2$. For a balanced spin-update engine, we will have $$\label{eq:balance1}
T_p = \lambda l^3 / F  \simeq T_c = 6 \iota l^2  / B\;,$$ where we have introduced average sustained processing power $F$ and average sustained interconnection bandwidth $B$ (both quantities can be very conveniently written in terms of operations (or bits exchanged) per processor clock cycle). Considering for definiteness the heat-bath algorithm, we have $\lambda \simeq 10$ (in $\lambda$ we have lumped a few short-word adds, the computation of one random number and access to one entry of a small look-up table), and $\iota =
1$. Comparing with the performance figures for the Pentium IV cluster, we have effective values of $F \simeq 0.3$, and $B \simeq 0.01$.

For the new machines, we use published data for $B$ and an approximate estimate for $F$ based on the assumption that the (clock-frequency normalized) performance of each integer processing unit available on these new processors is roughly the same as that available on a Pentium 4 (probably an upper limit, since these processors are all geared to floating point performance). We list relevant properties in table , where we also estimate the update time per site and the smallest value of $l$, $l_\mathrm{min}$, for which performance is not limited by communications, that we derive from Eq. ().

|        \-        |   Blue Gene    |       Cell       |      CSX600      |
|:----------------:|:--------------:|:----------------:|:----------------:|
|   Clock (GHZ)    |      0.7       |       3.2        |       0.25       |
|  $F$(ops/cycle)  |      0.3       |       2.4        |        32        |
|  $B$(bit/cycle)  |       12       |       192        |       256        |
| $l_\mathrm{min}$ |    $\ll 1$     |     $\ll 1$      |     $\ll 1$      |
|   Update Time    | $\simeq 45$ ns | $\simeq 1.25$ ns | $\simeq 1.33$ ns |

Architectural parameters for Blue Gene/L and for the IBM-Cell, and ClearSpeed CSX600 processors and their corresponding single-processor estimated spin-update time.

Note that these processors and systems are strongly optimized for floating-point calculations, not particularly useful in our context. This is worsened by the fact that fast floating point is single precision for Blue-Gene and the Cell processors (the latter also has less accurate rounding circuits) so its use for random number generation is questionable. For this reason sustainable $F$ values are smaller than would be achievable in floating point intensive computation. Inspection of table shows that the update-speed of one Blue-Gene/L node is smaller than on a conventional processor, while both the Cell and CSX600 processors promise a boost of almost one order of magnitude more than traditional high-end microprocessors. The real advantage potentially offered by all these new architectures however is that all the intrinsic parallelism of the underlying problem can be exploited without performance penalties.

# An overview of *Ianus*

*Ianus* is a massively parallel machine, optimized for the simulation of spin-glass systems and based on latest generation FPGA components.

In recent years, FPGA's have increased dramatically their processing speed and, above all, the logic complexity that they can incorporate. At present, logical systems of the order of millions of gates can be built inside just one FPGA device. At the same time, memory blocks of the order of millions of bits are built into high-end FPGA's. As already stressed, we expect to make much better than conventional processors in application domain, leveraging on three main cornerstones:

-   logic operations associated to spin-glass simulations differ from the arithmetic operations in which traditional computers focus.

-   our data-base can be fully contained by memories inside the processor. As a consequence an extremely high bandwidth is available to feed processing blocks.

-   FPGA's have very large resources for off-chip communication that we exploit for processor to processor data-transfer, with simple protocols that help reduce latency.

At the same time, FPGA's should help reduce development efforts, enlarging the window of opportunity for our machine, and allow, at a later stage, efficient tailoring of the developed hardware in different application areas.

In short, we are developing a *bi-programmable* machine, that we call *Ianus*. *Ianus* is based on the following main components:

-   A hardware layer, based on a module containing several high-end FPGA's.

-   A communication fabric that allows high bandwidth data exchange among the FPGA's and connection of the hardware module with a traditional host computer.

-   A software layer that allows to load a specific simulation model onto the hardware layer, set appropriate simulation parameters, run actual simulations and collect results.

-   At a later stage, we plan to add a model-development layer that allows to develop an ad-hoc FPGA based simulation engine for a large and growing class of interesting models with reasonable effort (we refer to this layer with the term bi-programmability).

The natural focus of *Ianus* will be toward discrete variables. Binary variables are the easiest example (and at first glance they can be seen as occupation variables, or as magnetic spins, or as boolean truth statements), but we plan and demand that *Ianus* should be as effective when dealing with $10$ or $50$ state variables. Typical fields we believe will be able to use *Ianus* effectively are:

-   Lattice models for physical systems.

-   Slow dynamics on simplified models: that is very important when discussing the former issue but is also very relevant for the next two issues.

-   Biological issues: Go models, statistical mechanics of DNA and RNA, docking.

-   Optimization and Cryptography problems.

# *Ianus* hardware

The hardware architecture of *Ianus* has already been developed: the basic computational block will contain 16 high-end FPGA chips (e.g., of the Altera Stratix or Xilinx Virtex families) that we call SP, (for Simulation Processors). We believe that using a powerful state-of-the-art core for the SP is an important feature of the machine we have in mind. Notice that each SP processor will be able to deal with of the order of $10^5$ logical elements at the same time (at least a $32^3$ lattice of Boolean variables when using binary link coupling).

The SP processors will be arranged and connected in a two dimensional array of $4\times 4$ processors: periodic boundary conditions will be enforced by hardware. A seventeenth processor will be used as an Input/Output Processor (IOP) for communicating with other boards. It will play the role of a cross-bar switch: each of the $16$ SP will have a (fast) link to the IOP. The IOP will perform two main functions, by allowing effective long range SP to SP communication and by acting as interface between the processing board and the host computer, a traditional Linux-based system.

In the long term, this hardware structure will be enabled by a growing set of FPGA-optimized building blocks (random number generators, Metropolis update engines, address-generators ...) to be used to make the SP's the dedicated/optimized processors for each target application. We also plan a user friendly framework interface that helps to assemble the wished set of computational functionalities, as well as a run-time support system that controls the configuration of the SP's and the actual simulation steps.

# Performance estimates for *Ianus*

We have already developed and tested preliminary versions of the FPGA based implementation of both the Demon and Heat-Bath algorithms. They are being tested on simple test boards using recent FPGA devices (Altera EP1S60 and Xilinx Virtex 4 LX25). For the final version of the machine we are considering either devices of the Altera Stratix EP2 or Xilinx Virtex 4 series. These more recent components have roughly up to 100% more available gates (and memory elements) and their speed is some 20% 30% higher. Our preliminary results are therefore lower limits on achievable performance.

We have developed and tested a simulating engine for the demon algorithm that tries to extract all possible parallelism. Our implementation of the algorithm is as follows: inside each system every site may be labeled as *black* or *white* following a checkerboard scheme, and each of the two subset may be updated in parallel, all neighbors of a black site being white ones. We define $K=N/2$ demon reservoirs, the first one of them coupled to the first black spin and the first white one and so on; then we proceed with parallel update within each set, alternating the simultaneous update of $N/2$ black spins and $N/2$ white ones. When simulating spin glasses one usually defines two identical replicas of the system, with same fixed $\left[J_{ij}\right]$ coupling configuration and different independent initial spin configurations; this leads to an immediate improvement of the algorithm: we put all black spins of replica 1 and all white spins of replica 2 in the respective positions of an "artificial" lattice system, say the *P lattice*, while in the *Q lattice* we put black spins of replica 2 together with white spins of replica 1. This way, each spin in P has its neighborhood in Q and vice versa. Also we double the number of demons to $K=N$. This way we can update in parallel a whole lattice of $N$ spins, the P or the Q one in turn.

Preliminary tests show that we need only a 5 ns clock cycle to perform the simultaneous update of up to 1000 spins, that corresponds to a lattice size $L=10$. In doing this, we only made use of flip-flop registers to store variables and the connectivity has been "hard-wired"; an "update engine" module is defined (taking as input a spin variable, its demon and neighborhood and outputting updated spin and demon) and the updating logic is replicated once for each site of the lattice; each variable register is directly connected to the appropriate ports of the appropriate engines. This approach is very efficient but pays the cost of being logic consuming: the "finiteness" of an FPGA strongly limits the maximum size $L$. This way we can simulate up to $L=10$ on an Altera Stratix EP1S60 and up to $L=6$ on a Xilinx Virtex 4 LX25.

In order to improve the size on these preliminary tests, we can hardwire connections only on one plane of the cubic lattice, implementing $K=L^2$ demons and updating engines, saving logic to store systems of size up to $L=10$ on the relatively small Xilinx Virtex4 xc4vlx25 device and up to $L=14$ on the Altera Stratix EP1S60. Even if this way we need $L$ clock cycles to update the whole lattice, we can update a spin every $0.1$. Also a wise use of memory blocks inside FPGA's guarantees raising sizes by a factor 3 and more, with a small cost in speed.

On the final machine, we can easily subdivide a $64\times 64\times 64$ system among the 16 SP's of the dedicated machine board in slices of $4\times
64\times 64$. Cautiously, we update a $4\times 1\times 64$ slice at a time, taking no more than 64 cycles to update in parallel all the system (a quarter of million spins!). Exploiting low latencies, the algorithm may be tailored to send computed boundary data (two faces of $64\times 64$ data bits) while the sequential update is running inside each SP. $B = 128$ would provide a sustained speed of $16 \times
4\times 64 = 4096$ spins per clock cycle, with a parallel efficiency of 100%. With a clock cycle of 5 nanoseconds, it corresponds to less than $1.3$ picoseconds per spin update.

For the Heat Bath algorithm we need a random number generator for each site that we want to update in parallel. We have implemented an FPGA version that includes 128 update engines, each equipped with its own random number generator (based on the shift register algorithm of , with 32-bit words). At present, for test purposes, we use a lattice of $32^3$ sites completely contained within the processor. The system updates one two-dimensional plane at a time. Using 128 update-engines in parallel, each plane is updated in 8 clock cycles, then the following planes are processed.

If we consider the same machine structure as above, and the same physical lattice, we may split the physical lattice in several ways. One possible solution, that minimizes bandwidth requirement has each processor handling a sublattice of $16 \times 16 \times 64$ lattice points. Our processors update $128 \times 16$ spin per clock cycles, corresponding to an update time of approximately $2.5$ ps. This performance is sustained by a bandwidth as small as 32 bits per clock cycle.

Extrapolating our preliminary results, we can conclude that *Ianus* (with, e.g., 16 processing boards) will be equivalent to a Cluster of $\simeq 5 \times 10^4$ Pentium IV processors or to a partition of $\simeq 3 \times 10^5$ Blue-Gene/L processors, or to an assembly of $\ge 8000$ Cell or Clearspeed processors.

At present we are controlling that the Random Generators we will use are correct, and also that the parallel updating scheme does not introduce larger correlations between configurations: eventually the plane-sequential version of the implementation should be safer. At the same time we are finalizing the FPGA design and starting to design the actual hardware infrastructure and the software structure needed to interface to traditional Linux-based host computers.

# Conclusions

Probably what is more important to stress here is that *Ianus* wants to be a machine that can solve a large number of problems. We believe that even models defined on random graphs, or where variables can take a finite number of values, will be very well suited for *Ianus*: problems like the study of Go models for protein folding and, more in general, biological issues, optimization issues and more will fit very well this pattern.

We already showed that it is possible to set up a computing device with an absolutely favorable performances-to-costs ratio, which is likely to be long-lived in the field of Ising spin glasses simulations. Keeping in mind that several topics in the field of complex systems are strictly related to the physics of Ising spin systems (among the classes of problems cited above, for example, error correction codes and the satisfiability of Boolean formulae), and that disposable technology is giving us many more possibilities with respect to the previous (and fruitful) SUE experience, we are confident in *Ianus* turning out as a "dedicated" and "multi-purpose" machine at the same time: another two-faced aspect justifying the choice of its name.

**Acknowledgments**

We wish to thank Piero Vicini, Alessandro Lonardo, Davide Rossetti and Sergio De Luca for very useful discussions. We also thank Liliana Arrachea, Pierpaolo Bruscolini, Kenneth Dawson, Giacomo Marchiori, Yamir Moreno Vega, Giorgio Parisi, Ilenia Pedron and Federico Ricci-Tersenghi for sharing with us interesting ideas. This work has been partially supported by DGA, MEC (BFM2003-C08532-C03, FIS04-5073-C04, FISES2004-01399, FPA04-2602) and the European Community's Human Potential Programme, contracts HPRN-CT-2002-00307 (DYGLAGEMEM) and HPRN-CT-2002-00319 (STIPCO).


abstract: This paper introduces a novel method for learning how to play the most difficult Atari 2600 games from the Arcade Learning Environment using deep reinforcement learning. The proposed method, called *human checkpoint replay*, consists in using checkpoints sampled from human gameplay as starting points for the learning process. This is meant to compensate for the difficulties of current exploration strategies, such as $\varepsilon$-greedy, to find successful control policies in games with sparse rewards. Like other deep reinforcement learning architectures, our model uses a convolutional neural network that receives only raw pixel inputs to estimate the state value function. We tested our method on Montezuma's Revenge and Private Eye, two of the most challenging games from the Atari platform. The results we obtained show a substantial improvement compared to previous learning approaches, as well as over a random player. We also propose a method for training deep reinforcement learning agents using human gameplay experience, which we call human experience replay.
author: Ionel-Alexandru Hosu; Traian Rebedea
bibliography: ecai.bib
title: Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay

# INTRODUCTION

General game playing is an extremely complex challenge, since building a model that is able to learn to play any game is a task that is closely related to achieving artificial general intelligence (AGI). Video games are an appropriate test bench for general purpose agents, since the wide variety of games allows solutions to use and hone many different skills like control, strategy, long term planning and so on. Designed to provide enough of a challenge to human players, Atari games in particular are a good testbed for incipient general intelligence. With the release of the Arcade Learning Environment (ALE) in 2012, general game playing started to gain more popularity. ALE is an emulator for the Atari 2600 gaming platform and it currently supports more than 50 Atari games. They are somewhat simple, but they provide high-dimensional sensory input through RGB images (game screen).

Deep learning models have achieved state-of-the-art results in domains such as vision and speech recognition . This is due to the ability of models such as convolutional neural networks to learn high-level features from large datasets. Along with these achievements, reinforcement learning also gained a lot of ground recently. These powerful models helped reinforcement learning where it struggled the most, by providing a more flexible state representation. As a consequence, tasks such as learning multiple Atari games using a single, unmodified architecture became achievable through deep reinforcement learning . However, in environments characterized by a sparse or delayed reward, reinforcement learning alone is still struggling. This is caused mostly by naive exploration strategies, such as $\varepsilon$-greedy , that fail to find successful policies to discover an incipient set of rewards. This is the case for the most difficult video games from the Atari platform, such as Montezuma's Revenge and Private Eye, that prove to be too challenging for all current approaches.

In a different context, AlphaGo , a system combining reinforcement learning with Monte Carlo Tree Search, defeated Lee Sedol, one of the top Go players in the world. Nearly 20 years after Garry Kasparov was defeated by Deep Blue , this represented an important milestone in the quest for achieving artificial general intelligence. Together with the launch of the OpenAI Gym , it is one of the most significant advances that reinforcement learning made in the last couple of years.

This paper demonstrates that it is possible to successfully use a learning approach on the most complex Atari video games, by introducing a method called *human checkpoint replay*. Our method consists of using checkpoints sampled from the gameplay of a human player as starting points for the training process of a convolutional neural network trained with deep reinforcement learning .

The paper proceeds as follows. In section 2, we present the most relevant results on learning for Atari video games. Section 3 follows with a more in-depth analysis and explanation of the current results for Atari games. We also motivate our choice of games for evaluating our architecture, providing a brief description of the difficulties encountered in these games. In section 4 we provide a thorough description of the proposed methods and deep reinforcement learning architecture. Section 5 presents the results of our approach, as well as a detailed discussion on the performed experiments.

# RELATED WORK

After the release of the Arcade Learning Environment, there have been numerous approaches to general game playing for Atari games. Approaches such as SARSA and contingency awareness delivered promising results, but were far from human-level performance. The use of neuro-evolution on the Atari platform dramatically improved these results, but playing Atari games as well as a human player still seemed unachievable.

The first method to achieve human-level performance in an Atari game is deep reinforcement learning . It mainly consists of a convolutional neural network trained using Q-learning with experience replay . The neural network receives four consecutive game screens, and outputs Q-values for each possible action in the game. Experience replay is used in order to break the correlations between consecutive updates, as Q-learning would prove unstable in an online setting. The most important aspect of this approach is that it can be used to construct agents that do not possess any prior domain knowledge, thus rendering them capable of learning to perform multiple different tasks.

After this first success of deep reinforcement learning , a number of improvements have been made to the original architecture. The fact that its convergence was slow and it took multiple days to train a neural network on a single game motivated the development of a distributed version of deep reinforcement learning which reduces the training times and improves the existing results.

Another notable improvement came from the realization that the Q-learning algorithm sometimes performs poorly by overestimating action values . This issue may be solved by employing double Q-learning  - using it together with deep reinforcement learning on the Atari domain fixed the overestimation problem that appeared in some of the games.

Another notable approach to Atari game playing is the bootstrapped deep Q-network (DQN) , which proposes a novel and computationally efficient method of exploration. Its main contribution is to find an alternative for simple, inefficient exploration strategies, such as $\varepsilon$-greedy. To achieve this, bootstrapped DQN produces distributions over Q-values instead of Q-values. Sampling from these distributions allows the model to renounce using exploration strategies.

The current state-of-the-art on Atari games is achieved using a method called prioritized experience replay . It starts from the assumption that not all the transitions present in the replay memory have the same importance. The agent learns more effectively from some transitions, other ones being redundant, not relevant, etc. The method proposes a prioritization regarding how often transitions are used for updates in the network based on the magnitude of their temporal difference (TD) error . Prioritized replay leads to an improvement in 41 out of the 49 games, delivering human-level performance in 35 of these games.

Although the aforementioned methods brought considerable improvements over the original deep reinforcement learning architecture, there still are some Atari games for which none of the previously published methods are able to learn whatsoever. These games feature a sparse reward space and are more complex in many aspects than the vast majority of games from the Atari platform. We propose a new method called *human checkpoint replay* that, when used together with deep reinforcement learning, is able to learn successful policies for the most difficult games from the Atari platform, thus resulting in significantly improved performance compared to prior work.

# BACKGROUND

Before proceeding to describe our approach, it is important to perform a more detailed analysis of the games used in our experiments; the discussion focuses on highlighting some of the aspects that make them so challenging. To evaluate our approach, we have chosen Montezuma's Revenge and Private Eye - two of the most difficult games from the Atari platform. Therefore our analysis focuses on these two games; however the main points are also valid for other challenging games where there is no known learning policy which achieves a better score than a random agent.

## Montezuma's Revenge

Montezuma's Revenge is a game from the Atari 2600 gaming console that features a human-like avatar moving across a series of 2D rooms that form labyrinth. In order to advance in the game, the player must move in a consistent manner, jump over obstacles, climb ladders, avoid or kill monsters, and collect keys, jewels and other artifacts which provide a positive reward by increasing the game score. Some of these collectible artifacts grant additional abilities - for example, collecting a key enables the player to open a door upon contact. However, after opening a door, the player loses the key and needs to collect additional keys for opening any other doors. Other collectible items include a torch which lights up dark rooms and swords which kill monsters. The game consists of three levels, each of them containing 24 rooms.

An important characteristic of the game is that collectible items are sparse and thus the reward space is also very sparse. When the game starts, in order to collect the first reward - represented by a key - the avatar is required to descend on two ladders, walk across the screen and over a suspended platform, jump over a monster and climb another ladder. After collecting the key, the avatar needs to return close to the starting point, where two doors are available which can now be opened using the collected key. The player starts the game with five "lives", and each time it loses one life, the avatar is respawned in the same room. For optimal play, a memory component is required, as the player does not possess information about other rooms in terms of rewards collected or monsters killed. The only information available on the screen apart from the environment is the game score, the number of remaining lives and the artifacts currently held by the avatar.

All current approaches using deep reinforcement learning fail to learn any successful control policies for Montezuma's Revenge. This happens mostly due to the $\varepsilon$-greedy strategy failing to explore the game in a consistent and efficient manner. Every four frames, a DQN agent has to choose between 18 different actions. Given how, in order to receive the first positive reward, the player is required to perform a complex and consistent sequence of actions, using such a simple exploration strategy makes learning virtually impossible. It is also worth mentioning that the way in which such a simple strategy explores an environment like Montezuma's Revenge does not resemble the way in which a human player does it. This is due to two factors. First, there is a strong correlation between multiple successive actions of a human player in the video game, even when the player does not know how to play the game yet. This is because the exploration exhibited by a human player is not random, but influenced by the current state of the environment. Second, a human player always makes use of commonsense knowledge when dealing with a new learning task. This also influences the manner in which the human player explores a virtual environment, especially when it contains elements resembling real-world objects (e.g. ladders, monsters, keys, doors, etc.) For example, when the game avatar is located on a ladder, a human player will only use the up and down actions without having to learn to do so by exploring the game states. The player already knows, from real life, that other actions are not useful, as they do not lead to other states in the game.

## Private Eye

The second game chosen for our evaluation is Private Eye. It is a game that features an avatar for a private investigator who is driving a car that can move around and jump vertically or over obstacles. The game environment can be seen as a labyrinth as well, as it consists of multiple roads located in a city or in the woods. In the city there are buildings near the roads, some of which play a special role in the game. The objective of the game is to capture thieves that sit behind the windows of different buildings. The thieves appear briefly and the player must move and "touch" the area near a thief in order to capture him/her. Capturing thieves provides the player with rewards and sometimes with special items, that must be returned to specific buildings (e.g. bags of money need to be returned to the bank, guns returned to the gun store). The player starts the game with 1000 points, but is penalized if he bumps into obstacles like birds and mice, or is attacked by thieves. The concept of multiple lives is not present in this game, however there is a three minute time cap for solving a game level; when the time limit has been reached, the game ends.

The game features similar exploration difficulties present in Montezuma's Revenge. As a consequence, this game has also proven to be a challenge for current deep reinforcement learning methods. The presence of a memory component for optimal play is even more important, as the player must travel long distances between collecting game items and dropping them at the appropriate locations. Some portions of the game look identical, and there is also a certain order in which the tasks should be carried out. For example, the game features a thief that must be captured and brought to the police building, but this can only be done after all the items in the current level have been returned where they belong. Capturing this final thief provides the highest in-game reward.

## Discussion

The more challenging games from the Atari console (such as Montezuma's Revenge and Private Eye) present a particular difficulty compared to the simpler ones: the agent is not penalized for standing still. This could be yet another factor that further prevents efficient exploration. In contrast, for games like Breakout or Pong (Figure ), repeatedly choosing the no-op action will quickly lead to the end of the game or to losing points. The agent can thus learn to avoid this action, as it will be associated with a low utility value.

# DEEP REINFORCEMENT LEARNING WITH HUMAN CHECKPOINT REPLAY

In this section we provide a description of the methods that we propose for training deep reinforcement learning agents on the Atari domain, as well as the architecture of the convolutional network that we used for the training process.

## Deep Reinforcement Learning

Deep reinforcement learning was the first method able to learn successful control policies directly from high-dimensional visual input on the Atari domain. It consists of a convolutional neural network that extracts features from the game frames and approximates the following action-value function

$$\begin{gathered}
\label{eq:state-action-value}
Q^{*}(a, b) = \\ \max_{\pi}\mathop{\mathrm{\mathbb{E}}}[r_{t} + \gamma r_{t+1} + \gamma^{2} r_{t+2} + ... \mid s_{t} = s, a_{t} = a, \pi]
\end{gathered}$$

The computed value represents the sum of rewards $r_t$ discounted by $\gamma$ at each time step $t$, using a policy $\pi$ for the observation $s$ and action $a$. To solve the instability issue that reinforcement learning presents when a neural network is used to approximate the state-value function, experience replay is used, as well as a target network . In order to train the network, Q-learning updates are applied on minibatches of experience, drawn at random from the replay memory. The Q-learning update at iteration $i$ uses the following loss function

$$\begin{gathered}
\label{eq:loss}
L_{i}(\theta_{i}) = \\ \mathop{\mathrm{\mathbb{E}}}_{(s, a, r, s') \sim U(D)} [(r + \gamma \max_{a'} Q(s', a'; \theta^{-}_{i}) - Q(s, a; \theta_{i}))^{2}]
\end{gathered}$$

Here $\gamma$ is the discount factor determining the agent's horizon, $\theta_i$ are the parameters of the Q-network at iteration $i$ and $\theta_i^-$ are the network parameters used to compute the target at iteration $i$. Differentiating the loss function with respect to the network weights gives the following gradient:

$$\begin{gathered}
\label{eq:gradient}
\nabla_{\theta_{i}} L(\theta_{i}) =  \mathop{\mathrm{\mathbb{E}}}_{s, a, r, s'} [(r + \gamma \max_{a'} Q(s', a'; \theta^{-}_{i}) - \\ - Q(s, a; \theta_{i}) \nabla_{\theta_{i}} Q(s, a; \theta_{i})].
\end{gathered}$$

Our proposed approaches use the deep reinforcement learning architecture from . Almost all the hyperparamaters have the same values, with the exception of the final exploration frame, for which we used a value of $4,000,000$ instead of $1,000,000$. We empirically found that using this value results in a slightly better performance of the agents. This may be caused by the fact that the most difficult games are also characterized by greater complexity and size of the state space, therefore a slower annealing of $\varepsilon$ may be helpful. Due to hardware limitations and the amount of time required for training deep Q-networks, we did not test for other values of this hyperparameter. Another structural difference we need to mention is the fact that, for the *human experience replay* method presented next, we employed an additional replay memory.

## Human Checkpoint Replay

For the most difficult games from the Atari platform that are characterized by sparse rewards, the original deep reinforcement learning approach is not able to achieve positive scores. Thus the agents trained using deep reinforcement learning perform no better than a random agent. As seen in the previous section, these games start in a state from which reaching the first reward is a long and challenging process for any player which does not possess any prior knowledge (such as commonsense world knowledge). Also considering the 18 actions available to the agent in each state, the $\varepsilon$-greedy strategy fails to find any game paths to a first state with positive reward. This hinders the convolutional neural network to learn relevant features to separate reward-winning states and actions from the bulk. Drawing inspiration from curriculum learning and the human starts evaluation metric used for testing Atari agents , we introduce the *human checkpoint replay* method. This consists of generating a number of checkpoints from human experience in the Arcade Learning Environment and storing them to be used as starting points for the training of deep reinforcement learning agents. Instead of resetting the environment to the beginning of the game each time a game episode ends, a checkpoint is randomly sampled from the checkpoint pool and restored in ALE.

The intuition behind this approach is that at least some of the checkpoints will have a reward located close enough in the game space for the $\varepsilon$-greedy strategy to be able to reach it. This way, the convolutional neural network is able to learn relevant features from the game frames and then successful control policies. As the training process advances, these will help the agent to become gradually more capable to reach rewards that are located farther away from the start state. Our method can also be thought of as being related to the planning concepts of landmarks and probabilistic roadmaps .

## Human Experience Replay

As a possible solution to the inefficiency of $\varepsilon$-greedy exploration in sparse reward environments, we also proposed training a deep reinforcement learning agent using offline human experience, combined with online agent experience. We dubbed this approach *human experience replay*. It consists of storing human gameplay experience in same form of $(s,a,r,s')$ tuples in a separate replay memory and using it along with the original replay memory containing agent experience. This is meant to provide the agent with training samples that result in a positive reward, therefore making learning possible in environments that feature a sparse reward signal. The training process consists of repeatedly sampling a minibatch composed of both human transitions and agent transitions.

# EXPERIMENTS

This section provides a thorough description of the experiments performed using our proposed approach for the two selected Atari games, Montezuma's Revenge and Private Eye. As mentioned earlier, we have chosen these two games because they are among the most challenging games on the Atari platform, even for human players. At this point no computer strategy has been able to learn an exploration technique which is better than a random agent, mainly due to the fact that no strategy is able to learn a solution which is able to reach any reward in the game. Therefore, any progress made towards solving these games will provides useful insights in the quest of developing general purpose agents.

## Human Checkpoint Replay

The Arcade Learning Environment provides the capability of generating checkpoints during gameplay. These make it possible to continue running an environment from a given state at a later time by restoring a specific checkpoint in the emulator. The checkpoint consists of the memory content of the Atari 2600 console.

For the human checkpoint replay method (HCR DQN), we generated 100 checkpoints from a human player's experience for each game, stored them in an external file and then used them as starting points for the environment at training time, as well as for testing. The checkpoints that we used for training and testing, as well as the code for training deep reinforcement learning agents with human checkpoint replay, are publicly available [^1]. We trained our networks using the generated checkpoints and performing Q-learning updates as describe in for 50 million frames on each game. The two plots in Figure show how the average predicted $Q$ evolves during training on the games Montezuma's Revenge and Private Eye.

As discussed in section , in difficult games such as Montezuma's Revenge and Private Eye, the avatar is not penalized for repeatedly choosing the no-op action. This raises two major issues. First of all, efficient exploration is prevented due to the neutral effect of repeatedly taking the no-op action. Also, in a deep reinforcement learning setting in which experience replay is used, by repeatedly choosing the no-op action, the replay memory will consistently be filled with transitions that are not relevant for the learning process. In order to avoid this outcome, we limit each training episode to 1800 frames, corresponding to 30 seconds of gameplay. By doing this, we make sure that the replay memory is populated with transitions that are relevant for the training process, as the agent will eventually be placed in checkpoints from which rewards are more easily accessible.

## Human Experience Replay

Using ALE, we generated 1.2 million frames of human experience (about 5.5 hours of gameplay) for Montezuma's Revenge, consisting of $(s,a,r,s')$ transition tuples. Human experience transitions were stored in an additional replay memory during training. We performed Q-learning updates for 15 million frames on minibatches of size 32, composed of 16 samples of human experience and 16 samples of online agent experience. Due to the long training times required for training deep Q-networks and the cumbersome process of generating multiple hours of human experience, we only tested this approach on Montezuma's Revenge.

## Evaluation procedure

In this paper, we used the human starts evaluation metric to test the performance of the agents. The metric consists of using random checkpoints sampled from human experience as starting points for the evaluation of an agent. More specifically, we use a set of 100 checkpoints as human-generated start frames. In order to prove the robustness of our agent, the set of checkpoints used for evaluation are different than the ones that were used for training. This evaluation method averages the score over 100 evaluations of 30 minutes of game time. The value of $\varepsilon$ was fixed to $0.05$ throughout the evaluation process.

The random agent's scores were obtained using the same evaluation procedure. However the next action to be performed in the environment was sampled from an uniform distribution.

## Quantitative Results

As it can be observed in Table , the human checkpoint replay method provides a substantial improvement over a random agent for both games. In Montezuma's Revenge it obtains more than double the points of a random agent. In Private Eye, a random agent is not able to obtain a positive score, due to the multitude of negative rewards present in the game which the agent is not able to avoid. Our HCR DQN agent obtains significantly better results, demonstrating the success of this approach.

Compared to the HCR DQN agent, the human experience replay method provides only slightly better performance over a random agent in Montezuma's Revenge. Due to sparsity of rewards during the game, human experience alone cannot provide enough transitions that lead to positive rewards in order to facilitate learning, although it does provide a slightly better exploration compared to the random agent.

|                     | Random Agent | **HCR DQN** | HER DQN |
|:--------------------|:------------:|:-----------:|:-------:|
|                     |              |             |         |
| Montezuma's Revenge |    177.1     |  **379.1**  |   218   |
|                     |              |             |         |
| Private Eye         |     –41      | **1264.4**  |   N/A   |

Results obtained by our methods, human checkpoint replay (HCR DQN) and human experience replay (HER DQN) on Montezuma's Revenge and Private Eye. The results represent raw game scores and were obtained using the human starts evaluation metric.

## Qualitative Results

We can draw better insights in the exploration of the agents by taking a closer look on the actions chosen by an agent. For Montezuma's Revenge, the HCR DQN agent is successfully collecting nearby rewards for all start points. For example, in the initial room of level 1, it successfully learns to climb the leftmost ladder in order to get the key. However, the agent still does not learn to avoid monsters and objects that lead to hypothetical negative rewards (such as losing a game life). This is mostly due to the fact that the game does not feature any negative rewards seen as changes in the score. This makes it even more difficult to find a successful exploration policy. While ALE offers this possibility, we did not provide our agents with an additional reward signal which penalizes the agent when it loses a life. It is also important to mention that in the majority of ALE checkpoints generated for training on Montezuma's Revenge there is no reward nearby. As a consequence, this set of checkpoints will continue to provide a challenge for future architectures, as it preserves much of the game's initial difficulty.

In Private Eye, the HCR DQN agent successfully collects most of the nearby rewards, and is seen successfully avoiding objects that lead to negative rewards. The agent does a great job at the latter task, especially as some of the negative rewards the avatar must avoid are moving fast and in an unpredictable manner, making this a difficult task even for human players.

## Discussion

Using human checkpoint replay might be seen as a trade-off for developing general game playing agents. The main objection would be that the agent uses human-generated start points for training the exploration model, which can be seen by some as an "deus ex-machina" intervention for the agent. However, the checkpoint replay merely provides additional starting points and does not offer an understanding of the explored game. It uses the experience of a human player to reach "easier" starting points, but for more difficult games this kind of intervention might be needed.

We should take into consideration that human players also make use of commonsense knowledge for these more difficult games. Using checkpoint replay does not bring any of this prior knowledge directly to the agent, maybe only in an indirect fashion as the human used it to get to that specific game positions.

# CONCLUSION

In this paper we presented a novel method using deep reinforcement learning, called human checkpoint replay, which was designed for some of the most difficult Atari 2600 games from the Arcade Learning Environment. Our experiments show a substantial improvement compared to all previous learning approaches, as well as over a random player. Our method draws inspiration from curriculum learning and it serves the purpose of compensating for the difficulties of current exploration strategies to find successful control policies in environments with sparse rewards.

As the results show, this method is a promising path of research. We will continue to study other approaches that deal with incentivizing and facilitating exploration in the most difficult games from the Atari platform. We believe that successfully learning control policies in such environments is closely related to the problem of achieving artificial general intelligence as in most real-life situations rewards are not encountered very frequently.

[^1]: Code and checkpoints are available here: https://github.com/ionelhosu/atari-human-checkpoint-replay


abstract: Here we will discuss the philosophical differences between an approach to the deep nature of physical space based on the concept of coordinates and one based on the concept of relation. The philosophical superiority of the second approach will be analysed and maintained, attempting to bring it to its extreme consequences. We will propose considering the concept of spatial dimension completely superfluous in favour of an idea of space as a pure relational structure, capable of including within itself the information necessary to define the properties of the things in the world.
author: Marcello Poletti
bibliography: Eng.bib
date: 2024-09-30
title: Space as relation

# Introduction

In his "Popular Exposition", published just a year after the fundamental articles of 1915, Einstein describes an attempt by Ernst Mach to solve the problem of inertia by rethinking space in a purely relational form, based on the concept of distance rather than position.

References to this possibility are found in many authors, from Poincaré to Rovelli and Vidotto.

Classical geometry typically studies the relationships between geometric entities such as points, segments, and lines. The concept of position (or coordinate) has been imposed since geometry became analytic with the famous works of Descartes.

Analytic geometry is capable of transforming properties of the pair of entities $\{A,B\}$ into atomic pseudo-properties of $A$ and $B$, forcing a kind of reductionism through the introduction of arbitrary properties and functions through which relational properties can be reconstructed.

In particular, the relational property "A and B are d apart" is transformed into the reduced properties:

-   $A$ has coordinates $A_i$

-   $B$ has coordinates $B_i$

From these properties is possible to reconstruct the distance $d$ through an appropriate function: $$\label{dist}
    d(A,B)=\sqrt{\sum{(A_i-B_i)^2}}$$ The properties $A_i$ and $B_i$ turn out to be completely arbitrary, and can be freely chosen within a series of transformations (rototranslations) that keep invariant; for all intents and purposes, these properties act as phases whose proper value is arbitrary and whose only concrete effect is to enter into a relation (typically a difference) with other phases of the same type.

The function selected for the distance is such as to automatically guarantee that triangular inequalities are respected for every possible choice of coordinates, and this is extraordinarily effective in freeing the geometry from annoying constructability constraints.

This approach, which we will call *positional* here, introduces an additional fundamental parameter: the number of coordinates to be used for each point, the *number of dimensions*.

This value is equally arbitrary in the reconstruction of the distance $(A,B)$, while it assumes a central role in the case of considering more than just two points. In particular, this value plays the role of a constraint with respect to the possible distance values allowed; in an m-dimensional space for $m+1$ points only, it is possible to define every possible combination of distance, respecting the triangular inequalities; additional points will be (automatically) constrained so that some distances, even if compatible with the triangular inequalities, are not allowed.

Alternatively, it is possible to define the geometry of $n$ points through their reciprocal $n(n-1)/2$ distance relations, subject to the complex constraints given by the triangular inequalities. This approach, which we will call *relational*, presents no symmetries and has no dimension parameters. The relational approach is **not** equivalent to the positional approach, because the constraints imposed by dimensionality do not apply to it. In a relational geometry, by definition, **every possible distance is allowed, respecting only the triangular inequalities**. In this sense, the relational approach is broader than the positional one.

# Non-Euclidean Geometry

In non-Euclidean geometry, the positional approach undergoes important updates.

In addition to $A_i$ coordinates, the concept of metric tensor $g_{ab}$ is introduced, defining the shape of the space through the fundamental relation which extends the $$ds^2=g_{ab}dx^adx^b$$ The action of $g_{ab}$ is to make relationships of distance more elastic by reducing the role of the constraints given by the dimensions; a curved variety allows distances typically not definable in a Euclidean space.

It is natural to wonder to exactly what extent these constraints are reduced. The answer, curiously rarely emphasized in geometry textbooks, is that such constraints disappear altogether in spaces of dimension 3 or higher \[Appendix \].

The non-Euclidean positional approach, in relation to reciprocal distances, is equivalent to the relational approach.

Viewed retrospectively, then, non-Euclidean geometry is none other than a way to free the positional approach from its dimensional constraints. The positional approach is finally equivalent to the relational approach for varieties of dimension at least 3. Conversely, the relational approach spontaneously conduces to non-Euclidean spaces of dimension at least 3; Mach's intuition was therefore correct: the relational approach has in itself properties equivalent to the concept of curvature and it was precisely these properties that led Einstein to the solution of the problem of inertia.

At this point, however, one wonders if the advantages introduced by the positional approach, particularly the automatic validity of triangular inequalities, continues to outweigh its costs, in particular:

-   The complexity presented by the $g_{ab}$ tensor

-   The presence of gauges due to the arbitrariness of the coordinates/phases.

-   The presence of the ambiguous parameter related to the number of dimensions.

# Metric spaces

The formalization of the positional approach leads to the concept of normed vector space; the formalization of the relational approach leads, in the first instance, to the weaker concept of metric space.

On the one hand, vector spaces have shown themselves to be extraordinarily effective in a great many areas even beyond the modelling of physical space; on the other hand, metric spaces turn out to be too poor as possible substitutes. In particular, the former allow us to easily define all essential geometric sizes, angles, areas, volumes, etc., while the latter do not.

A metric space is essentially a set over which distances are defined, respecting triangular inequalities. This definition is ultimately too weak, as it does not include the concept of path or beetwinnes.

Intuitively, what characterizes the distance relation in physical space between points $A$ and $B$ is precisely the existence of paths between $A$ and $B$, "distance" emerging a posteriori as a search for the minimum of the path lengths. In simple terms, $A$ and $B$ are 6 steps apart in the sense that it is, at a minimum, possible to travel 6 steps to get from $A$ to $B$, and this implies that point C also exists at step 1, point $D$ at step 2, and so on.

Since metric spaces do not contain the concept of path, they do not contain the concept of topology, nor the concepts of angle, area, volume, etc.

There are many alternatives oriented toward solving this difficulty: Alexandrov spaces, length spaces, topological spaces, angle spaces, proximity spaces, etc.

These last are particularly attractive with respect to instances of natural philosophy, basing the spatial relationship on the classical concept of locality: $A$ is proximate to $B$, or adjacent to $B$ or, more simply near $B$, while $C$ is distant and reachable through a chain of proximity.

# Proximity spaces

A proximity space $(X,\rho)$ is defined as a set $X$ equipped with a function with domain in $X$ and co-domain on the set of parts of $X$, $\rho(X)\to\mathcal{P}(X)$, such that $$\begin{cases}
        A\notin\rho(A) \\
        A\in\rho(B)\iff B\in\rho(A)
    \end{cases}$$ This definition of proximity space is remarkably simple, inherently discrete, and entirely sufficient for the purposes of the reflections made here.

A proximity chain is defined as a succession of points $\{P_i\}$ such that $P_i\in\rho(P_{i+1})$, empty successions or successions containing only one point are conventionally included.

A path between $A$ and $B$ is defined as a finite proximity chain that starts at $A$ and ends at $B$. $$P^A_B = \begin{cases}
        \{P_i\}_{i=1}^n \\
        P_1=A \\
        P_n=B
    \end{cases}$$ The length of the path between $A$ and $B$ is defined as the quantity: $$|P^A_B| = Card(\{P_i\}_{i=1}^n)-1=n-1$$ The proximity space $(X,\rho)$ will be considered complete if, for each pair of points $A$, $B$, there exists a path between $A$ and $B$; it will be considered finite if $Card(X)<\aleph_0$ and it will be considered locally finite if $\forall A\in X,Card(\rho(A))<\aleph_0$

On a proximity space it turns out that the metric is clearly defined: $$d(A,B)=min(|P_A^B |)$$ Note how in a proximity space the concept of distance is in a certain sense accidental: distance is a minimum of a larger condition, the length of the paths, and this complex definition leads, again accidentally, to the condition of triangular inequalities.

A finite proximity space substantially coincides with a graph on which the metric induced by the minimal paths is defined; a locally finite proximity space on the other hand has the appearance of a possibly infinitely extended graph.

# Dimensions

There are several cutting-edge theories that take an approach to the nature of the space in various aspects in line with what is discussed here; we think in particular of LQG, Causal Sets, and the Wolfram Physic Project (WPP).

In all these cases, however, we observe a forcing to make the chosen structure "adapt" to a parameter that is itself absent in the base setting, the number of dimensions.

In the WPP manifesto we read, for example:

> *I'm frankly amazed at how much we've been able to figure out just from the general structure of our models. But to get a final fundamental theory of physics we've still got to find a specific rule. A rule that gives us 3 (or so) dimensions of space, …*

And Fay Dowker, writes, in relation to Causal Sets:

> *If spacetime is a causet, however, coarse graining it at different scales (i.e. with different deletion probabilities p) gives rise to different causets which may have continuum approximations with different topologies, including different dimensions.*

The impression is that these authors are searching for a relational space which, at least at a certain scale, adequately emulates a three-dimensional positional approach, but as we can see, this is essentially useless. The number of dimensions can be considered entirely superabundant, emergent if you will, an unwelcome remnant of the positional approach, a necessary cost of vector formalism and its extraordinary effectiveness.

The relational approach under discussion does not indicate such a parameter, yet it is quite evident that this parameter has a role in physical space: it is possible to arrange three pencils so that they are mutually perpendicular, and it is not possible to do it with four.

One possible solution to this paradox is that the concept of "pencil" be closely related to the metric properties themselves, and this is exactly what happens in general relativity.

Wheeler wrote:

> *Could the metric continuum be a magical medium, which folded here in a given way represents a gravitational field, waved there in another way describes an electromagnetic field, and twisted locally describes a persistent concentration of mass energy?*

This question is partially answered by Einstein's field equations: $$R_{\mu\nu}-\frac{1}{2}g_{\mu\nu}+\Lambda Rg_{\mu\nu}=\frac{8\pi G}{c^4}T_{\mu\nu}$$ which, usually read as "matter bends space-time", can just as well be read as "matter IS a fold of space-time".

If "pencil" is by definition the curvature of the 3D categorization of an inherently dimensionless space, then the paradox dissolves, "pencil" is a 3D object by definition.

This approach also allows us to reread some overdimensional theories from a new perspective. In particular, the Kaluza hypothesis can be considered completely legitimate in this perspective, redefining the electromagnetic field as a categorization of the same space in 4D, responding positively to Wheeler's provocation for this field as well, and without any need to assume "compactifications" of the exceeding dimension, such spaces being purely emergent, categorizations of an underlying dimensionless space.

# Conclusions

The historical and philosophical analysis of the concept of space conducted here compares the concept of Cartesian space - container of geometric entities - with the temptation that emerged many times in the history of scientific thought: giving up a containing background in favour of a geometry based purely on reciprocal relations.

From this emerges an idea of space that leads in the direction of Wheeler's fascinating proposal, a space that itself defines its content as a function of its own metric, which defines the real as the emergence of metric properties.

In a recent video, David Gross said

> *"...there are two facts that we know about nature: one is there's matter, like us..."*

The approach discussed here denies this statement, relegating matter to a pure 3D categorization of these properties.

Space - even in its traditional Euclidean view - is a purely relational structure, in which a marvellous recursion makes it so that the related entities within it, the points, have no property except to be able to enter into relation with other points. This structure is capable of holding local information by itself, encoded in the metric that is created (GR).

Note the philosophical difference between a vector of bits, objects capable of "containing" a value $\{0,1\}$ and implicitly ordered (otherwise the information contained in it is null), and the structure under discussion (for simplicity think of a graph) in which there is "nothing added" to the state of proximity of the points.

In this fascinating scenario, it seems necessary to also give up that parameter of dimension, so central to the reflections within physics of the last 50 years, to relegate it to a pure emergent value.

From the ontological perspective, it is frankly difficult to thoroughly estimate how much what is discussed has to do with "what is" and how much has instead to do with "what describes what is", but even in this weaker second version, I believe that the relational approach can be a mental beacon of reference. This idea essentially endorses the LQG, Causal Set, WPP-type approaches to the new physics (which not coincidentally exploit graph theory in one way or another), inviting promoters to avoid forcing - such as the search for particularly ad hoc spaces or the introduction of additional parameters (the addition of matter, the addition of properties of points, the addition of properties on the proximity, etc.) - by treating the spatial structure itself as a container for the necessary information by acting purely on the metric properties, following Wheeler's proposal and provocation to the end.

#

Let us consider $n$ points and their $n(n-1)/2$ freely chosen distances, respecting only the triangular inequalities. Let $d$ be the maximum of such distance.

Now arbitrarily choose $n$ points in a Euclidean space inside a sphere of diameter less than $d$ and connect each pair with a path of length equal to the distance, so that two paths never cross. This is always possible in at least 3D space.

Then deform the metric at each point that does not belong to a path, so that these paths become geodesics. The manifold finally obtained contains the metric of the $n$ starting points.