abstract: Generation of photon pairs from compact, manufacturable and inexpensive silicon (Si) photonic devices at room temperature may help develop practical applications of quantum photonics. An important characteristic of photon-pair generation is the two-photon joint spectral intensity (JSI), which describes the frequency correlations of the photon pair. In particular, heralded single-photon generation requires uncorrelated photons, rather than the highly anti-correlated photons conventionally obtained under continuous-wave (CW) pumping. Recent attempts to achieve such a factorizable JSI have used short optical pulses from mode-locked lasers, which are much more expensive and bigger table-top or rack-sized instruments compared to the Si microchip pair generator, dominate the cost and inhibit the miniaturization of the source. Here, we generate photon pairs from a Si microring resonator by using an electronic step-recovery diode to drive an electro-optic modulator which carves the pump light from a CW optical diode into pulses of the appropriate width, thus potentially eliminating the need for optical mode-locked lasers.
author: Marc Savanier; Shayan Mookherjea
date: 2024-10-01
title: Generating photon pairs from a silicon microring resonator using an electronic step recovery diode for pump pulse generation

Room-temperature photon-pair and heralded single photon sources near $1.55\ \mathrm{\mu m}$ wavelength have traditionally been made using nonlinear crystals or optical fibers. Silicon (Si) photonic devices can also be used for photon-pair generation based on the spontaneous four-wave mixing (SFWM) nonlinearity. Unlike quantum dots, these photon sources can operate at room temperature and can be coupled relatively easily to optical fibers. The pump and generated photon wavelengths all lie within the telecommunications spectral windows used in fiber-optic networks. Moreover, such devices can nowadays be manufactured using CMOS-compatible processes on Si-on-insulator (SOI) and Si wafers, thus lowering the cost and increasing the likelihood for widespread applications.

It has been shown that photon pairs can be created in simple-to-fabricate structures such as Si microring resonators at a rate of a few hundred kilohertz, using approximately a milliwatt of optical pump power, based on the process of SFWM. Although both Si waveguides and micro-resonators have been used to create photon pair-generation devices, resonators use the optical pump power more efficiently than waveguides and they have generated photon pairs with a higher coincidence-to-accidentals (CAR) ratio. The quality factors ($Q$) of Si microrings used for pair generation have recently improved, but are generally lower than silicon nitride or glass microrings. However, silicon microrings have other benefits, including a compact footprint, a larger free-spectral range with fewer pairs of spectral lines being generated (since most applications today require only a single pair of photons), and since the pair-generation wavelengths shift with temperature, an electronic $p$-$i$-$n$ diode can be incorporated in the cross-section of the Si microring itself for real-time monitoring.

SFWM consists in the generation of two photons at radial frequencies $\omega_1$ and $\omega_2$ from two pump photons at $\omega_p$, with respective wavelengths $\lambda_i$ where $i = \{1, 2, p\}$, and energy conservation reading $2 \omega_p = \omega_1 + \omega_2$. In a microring resonator, its efficiency is maximum when the triple resonance condition is also fulfilled.

If a continuous-wave (CW) optical pump is used, the photon pairs that are generated by spontaneous nonlinear processes are highly correlated in the frequency-time domain, and can be entangled–as verified by two-photon interference measurements. The generated bi-photon spectrum, plotted in the ($\lambda_1, \lambda_2$) space and known as the joint spectral intensity (JSI) of the photon pair is generally elliptical (or multi-peaked), with a Schmidt number significantly greater than 1. When a heralded single-photon source is desired, it is necessary that the JSI be circular, which leads to a separable two-photon wavefunction. The width along the anti-diagonal axis in Fig. (a) (top-left to bottom-right) is determined by the product of the microring resonances at the generated photons wavelengths. The width of the JSI along the diagonal axis (from bottom-left to top-right) is determined by the convolution of the microring resonance at the pump wavelength and the pump pulse spectrum. Therefore, to achieve a circular JSI, we need to match the pump pulse duration to the resonance photon lifetime.

One approach is to use mode-locked lasers which generate short pulses optically; however, these are large, expensive and individually-assembled laboratory instruments, and reduce the level of integration, scalability and cost-advantages of using integrated photonics. It would be preferable to obtain pulses from a CW laser diode using a standard amplitude electro-optic modulator driven by a compact, low-cost electrical pulse generator.

*Device:* Our device, shown in Fig.  was fabricated at IME (Singapore) using CMOS-compatible process on standard SOI wafers widely used in Si photonics, with 220 nm Si thickness and $3\ \mathrm{\mu m}$ buried oxide thickness. Two etch steps were used to give a rib waveguide cross-section of height 220 nm, a waveguide width of 650 nm, and a slab thickness of 70 nm. The microring resonator had a radius of $R$ = 10 $\mu$m with inter-waveguide gap of 200 nm (satisfying the minimum feature-size recommendation for the foundry fabrication process). Only the lowest-order transverse electric (TE) polarized mode family propagates in such microrings. The $p$ and $n$ doped regions were formed by implanting Boron and Phosphorous, respectively, with the edge of the implanted regions being 900 nm from the edge of the ridge. These implanted regions create a photodiode which monitors the optical power in the microring, so that the pump wavelength can be aligned to the microring resonance despite temperature variations, as discussed elsewhere. Test waveguides similar to those used to form the microring resonator were measured using an atomic force microscope to have a root-mean-squared sidewall roughness of 2.6 nm, and, using an optical cutback method, to have a propagation loss of -0.74 dB/cm at 1550 nm. Using the rib waveguide cross-section, we can achieve a more favorable group velocity dispersion (GVD) than the fully-etched Si waveguide. We calculated a GVD coefficient of -1040 ps/nm-km at a wavelength of 1550 nm, whereas the "standard" fully-etched Si waveguide, with a similar cross-section has a much higher group-velocity dispersion (4400 ps/nm-km), as well as a higher optical propagation loss (typically 2–4 dB/cm), both of which can negatively impact pair generation. Fig. (b) suggests that we should continue to attempt to further lower the GVD from what we report here.

*Pulse generation:* Here, we study photon pairs generated by a silicon microring device, pumped optically by CW light from a laser diode which is carved into pulses using an off-the-shelf electro-optic amplitude modulator driven by a compact and inexpensive step-recovery diode (SRD). An SRD is a two-terminal $p$-$i$-$n$ diode whose junction impedance depends upon its internal charge storage. Fig. (d) sketches the operating principle of the SRD. Under forward-bias, the junction has low impedance and charge build-up occurs. When driven by the portion of the sinusoidal RF drive which reverse-biases the diode, the junction remains low-impedance until the stored charge is depleted. This condition abruptly causes the impedance to rise, stopping the flow of current. The forward bias current $i_F$, the equivalent resistance seen by the diode $R_\text{eq} = R/2$, the desired peak voltage at the output $V_L$ and the minority carrier lifetime $\tau$ determine the charge storage time $T_S \approx (i_F R_\text{eq} /V_L) \tau \approx 0.5~\text{ns}$ in our experiment. The voltage pulse is generated during the (much shorter) rise ("recovery") time of the diode current ($T_R$), which is determined by $R_\text{eq}$ and the diode junction capacitance $C_J$, so that $T_R \approx 3.1 R_\text{eq} C_J$, i.e., generating a pulse of duration 70 ps requires $C_J \approx 1$ pF. Although we use a commercial SRD module in this demonstration (Herotek GIM 1000A; bias current approximately 200 mA at 5 V DC bias, generating 9 V peak-voltage pulses of duration 63 ps into a 50 $\Omega$ load), such diodes can be fabricated on a chip, and can generate pulses of a few tens of picoseconds, if care is taken to minimize parasitic inductances and capacitances which limit the voltage slew-rate.

It is important to note that the specifications of the SRD guide the design of the Si microring resonator, and vice-versa. The relatively low-Q (unloaded quality factor $Q_U<1 \times 10^5$) Si microrings used for pair generation until recently require pulses that are only a few picoseconds in duration, and simple all-electronic pulse generation approaches are impossible. On the other hand, very high-Q Si micro-resonators can be fabricated, but create challenges in stabilization of the pair generation rate (PGR) and fabrication yield, because a phase-matching 'sinc' function comes into play, and small fabrication imperfections or environmental variations can cause the PGR to oscillate between 0 and its design value. We operate in the intermediate sweet-spot ($Q_U \approx 1 \times 10^6$), where the microring-waveguide coupling coefficient is chosen such that the loaded-cavity ($Q_L \approx 1 \times 10^5$) ringdown time is more or less matched to the SRD pulsewidth.

*Experiment:* The microresonator was optically pumped in the usual experimental setup for SFWM. An optical CW diode laser was carved into optical pulses at a repetition rate of 1 GHz using an off-the-shelf, fiber-pigtailed high bandwidth (40 GHz) amplitude electro-optic (EO) lithium-niobate modulator. To achieve a high extinction ratio at the modulator \[see Fig. (c)\], the EO modulator bias voltage was continuously controlled using a micro-electronic circuit based on a small pilot tone at the null-mode operating point. The RF input to the modulator was generated by the SRD. Based on the measured optical linewidth of the microring resonance, the photon lifetime was estimated to be $79\ \mathrm{ps}$. An SRD with a slightly shorter pulsewidth was used to drive the EO modulator \[see Fig. (c)\], allowing for a slight pulse broadening through the fibers and feeder waveguide.

Based on a separate calibration measurement, the fiber-to-waveguide optical insertion loss was estimated to be -3.5 dB at each facet. Photons at the output of the chip were spectrally separated using a custom-built tunable filter assembly (using telecommunications components) that suppressed the residual pump photons by more than 150 dB and extracted the signal and idler wavelengths, with insertion loss -6 dB and -5 dB, respectively. Single photons at the signal and idler wavelengths were detected using gated thermo-electrically cooled (233 K) InGaAs SPADs with an estimated 10$\%$ quantum efficiency, electrically-generated gate width of 3.0 ns and gating repetition frequency 60 MHz . The dark counts of the two SPADs were 1.4 kHz and 3.0 kHz. A time-correlated single photon counting (TCSPC) board was used to measure histograms of coincidence and accidental counts. The SPAD controller and the RF oscillator used to drive the SRD were sychronized at 10 MHz. The gate window at the SPADs accommodates 3 pump pulse time bins, each of which has a low generation probability $\approx$ 5.1 $\times$ 10$^{-5}$ pairs/pulse. Under these operating conditions, at a temperature of 30$^\circ$C and average input pump power (in the feeder waveguide) of 1 dBm at 1562.209 nm, the photon pairs at 1542.63 nm and 1582.26 nm were generated at a rate of about 733 kHz.

*JSI measurement:* Measurement of the JSI shown in Fig.  is time-consuming, and required the acquisition of 225 histograms of the start-stop coincidence inter-arrival times, each integrated for 60 seconds. (A classical seeded four-wave mixing experiment more rapidly measures the phase-matching properties inherent to the spontaneous mixing process, although not directly measuring the device when it is generating photon pairs.) To acquire the JSI more rapidly, we operated with a higher average pump power than we usually do; at lower pump powers, a $\text{CAR} \sim 100$ has been measured from the same microrings. The optical intensity enhancement factor (ratio of the circulating field intensity in the microring to the intensity in the feeder waveguide) was 346. Thus, an input waveguide power of 1 dBm resulted in a circulating field intensity in the microring of about $240\ \mathrm{MW}.\mathrm{cm}^{-2}$. Under these conditions, the associated two-photon absorption in silicon was calculated to be 0.5 dB/cm, i.e. slightly smaller than the linear propagation loss. In addition, from the higher pump power, a wavelength red-shift of 48 pm was observed, relative to the lower average pump power results we reported earlier.

The measured "raw" JSI represents the convolution of the actual JSI and the separately-measured (static) point-spread function of the filters. Similar to our previous work, we used the damped Richardson-Lucy (RL) algorithm, an iterative procedure for recovering, in a maximum-likelihood sense, a latent image that has been blurred by a known point spread function (PSF). The end-point of the RL iteration generally needs to be determined by the user, and we confirm our choice by a classical (seeded) four-wave mixing experiment on the same device. Using 25 iterations and a damping value of 1 standard-deviation (which are fairly typical values), the result is shown in Fig. (a). The 'raw' blurred JSI is shown in Fig. (b), and the PSF of the optical filters in Fig. (c). The PSF of these filters can be measured very accurately using a tunable CW laser and high dynamic range photodetector. The reconstruction does not show significant sensitivity to the exact number of iterations, or the value of the damping parameter. We used a sub-sampling procedure (by 8x) during reconstruction, which yields better positional fidelity when the PSF is well known, and results in a more accurate determination of lateral shifts of the reconstructed image in the plane. This allowed us to confirm the consequence of the wavelength red-shift discussed above in causing the experimentally-measured JSI \[Fig. (a)\] to move to longer wavelengths (towards the upper-right corner) compared to the simulated JSI \[Fig. (b)\], which ignored this thermal effect in the calculation.

*Schmidt decomposition:* A quantitative measure of the factorizability of the generated two-photon states is obtained by performing a Schmidt decomposition. A bi-partite state is separable (and suitable for heralding) if the number of non-zero eigenvalues is one. A widely-used operational measure for entanglement is therefore obtained by calculating the Schmidt number $K$, representing the number of orthogonal modes in the singular-value decomposition (SVD) of the magnitude of the joint spectral amplitude (JSA, calculated here as the square root of the measured JSI under the assumption of flat spectral phase). For the measured JSI shown in Fig. (a), we calculate $K = 1.12$, whereas the theoretically-expected value from Fig. (d) is $K = 1.21$. This is already a good level of agreement; furthermore, the simulation relies on the assumption of a specific waveguide height and width, whereas the fabricated waveguide inevitably had slightly different dimensions, which could alter the JSI as shown in the inset to Fig. (b).

Fig. (d) shows the distribution of the magnitudes of the first few singular values of the measured (deconvolved) JSI. This shows that the first mode dominates the distribution, although the contribution of the other modes is not zero. The shapes of the first three mutually-orthogonal spectral basis functions that form the singular value decomposition (SVD) of the JSA, are shown in Fig. (e)-(g), and as expected, show the symmetrical or anti-symmetrical behavior about the spectral center-point. Elimination of the contribution of the higher-order modes can be acheived by reducing the degree of spectral asymmetry shown in Fig. (a) between the pump (thick dashed white arrow line) and the center spot of the dual photon resonance.

*Summary:* Previously, generating close-to-circular JSI's with a low Schmidt number required precision poling of nonlinear crystals or pump beam shaping by bulk optics components, and the use of mode-locked lasers which generate short optical pulses. Here, the SRD solves the problem of pump-pulse generation in a compact footprint, and requires only a sinusoidal electronic drive waveform. The remaining impediment towards a completely circular JSI is the waveguide dispersion, which is quite large in conventional Si photonic structures, but can be improved with a new round of fabrication. More generally, our work demonstrates that Si microfabrication can be used to create simple microring based room-temperature optically-pumped pair generators. In the future, we anticipate that the Si micro-electronic components and the electro-optic modulator, and perhaps also the laser diode itself, can all be co-integrated with the Si microring resonator onto a monolithic platform, for a truly integrated photon-pair source.

The authors are grateful to R. Kumar (Intel Research, Santa Clara, CA), J. R. Ong (Institute of High Performance Computing, ASTAR, Singapore), Xianshu Luo and Guo-Qiang Patrick Lo (Institute of Microelectronics, ASTAR, Singapore), Nick Bertone (Optoelectronic Components), and the National Science Foundation for support (ECCS 1201308).


abstract: Pylearn2 is a machine learning research library. This does not just mean that it is a collection of machine learning algorithms that share a common API; it means that it has been designed for flexibility and extensibility in order to facilitate research projects that involve new or unusual use cases. In this paper we give a brief history of the library, an overview of its basic philosophy, a summary of the library's architecture, and a description of how the Pylearn2 community functions socially.
author: Ian J. Goodfellow; David Warde-Farley; Pascal Lamblin; Vincent Dumoulin; Mehdi Mirza; Razvan Pascanu; James Bergstra; Frédéric Bastien; Yoshua Bengio
bibliography: strings.bib; strings-shorter.bib; ml.bib; aigaion-shorter.bib; aigaion.bib
title: Pylearn2: a machine learning research library

# Introduction

Pylearn2 is a machine learning research library developed by LISA at Université de Montréal. The goal of the library is to facilitate machine learning research. This means that the library has a focus on flexibility and extensibility, in order to make sure that nearly any research idea is feasible to implement in the library. The target user base is machine learning researchers. Being "user friendly" for a research user means that it should be easy to understand exactly what the code is doing and configure it very precisely for any desired experiment. Sometimes this may come at the cost of requiring the user to be an expert practitioner, who must understand how the algorithm works in order to accomplish basic data analysis tasks. This is different from other notable machine learning libraries, such as scikit-learn  or the learning algorithms provided as part of OpenCV , the STAIR Vision Library , etc. Such machine learning libraries aim to provide good performance to users who do not necessarily understand how the underlying algorithm works. Pylearn2 has a different user base, and thus different design goals.

In this paper, we give a general sense of the library's design and how the community functions. We begin with a brief history of the library. Finally, we give an overview of the library's philosophy, the architecture of the library itself, and the development workflow that the Pylearn2 community uses to improve the library.

|  |  |
|:--:|:--:|
| GitHub repository | <https://github.com/lisa-lab/pylearn2> |
| Documentation | <http://deeplearning.net/software/pylearn2> |
| User mailing list | email@example.com |
| Developer mailing list | email@example.com |

Other Pylearn2 resources

# History

Pylearn2 is LISA's third major effort to design a flexible machine learning research library, the former two being PLearn and Pylearn. It is built on top of the lab's mathematical expression compiler, Theano . In late 2010, a series of committees of LISA lab members met to plan how to fulfill LISA's software development needs. These committees determined that no existing publicly available machine library had design goals that would satisfy the requirements imposed by the kind of research done at LISA. The committees decided to create Pylearn2, and drafted some basic design ideas and the guiding philosophy of the library. The first implementation work on Pylearn2 began as a class project in early 2011. The library was used for research work mostly within LISA over the next two years. In this time the structure of the library changed several times but eventually became stable. The addition of Continuous Integration from Travis-CI  with the development workflow from GitHub  helped to greatly improve the stability of the library. GitHub provides a useful interface for reviewing code before it is merged, and Travis-CI tells reviewers whether the code passes the tests.

In late 2011 Pylearn2 was used to win a transfer learning contest . After this, a handful of researchers outside LISA became interested in using it to reproduce the results from this challenge. However, the majority of Pylearn2 users were still LISA members. Pylearn2 first gained a significant user base outside LISA in the first half of 2013. This was in part due to the attention the library received after it was used to set the state of the art on several computer vision benchmark tasks , and in part due to many Kagglers starting to use the library after the baseline solutions to some Kaggle contests were provided in Pylearn2 format .

Today, over 250 GitHub users watch the repository, nearly 200 subscribe to the mailing list, and over 100 have made their own fork to work on new features. Over 30 GitHub users have contributed to the library.

# License and citation information

Pylearn2 is released under the 3-claused BSD license, so it may be used for commercial purposes. The license does not require anyone to cite Pylearn2, but if you use Pylearn2 in published research work we encourage you to cite this article.

# Philosophy

Development of Pylearn2 is guided by several principles:

-   **Pylearn2 is a machine learning research library–its users are researchers.** This means the Pylearn2 framework should not impose many restrictions on what is possible to do with the library, and it is acceptable to assume that the user has some technical sophistication and knowledge of machine learning.

-   **Pylearn2 is built from re-usable parts, that can be used in many combinations or independently.** In particular, no user should be forced to learn all parts of the library. If a user wants only to use a Pylearn2 `Model`, it should be possible to do so without learning about Pylearn2 `TrainingAlgorithm`s, `Cost`s, etc.

-   **Pylearn2 avoids over-planning.** Each feature is designed with an eye toward allowing more features to be developed in the future. We do enough planning to ensure that our designs are modular and easy to extend. We generally do not do much more planning than that. This avoids paralysis and over-engineering.

-   **Pylearn2 provides a domain-specific language that provides a compact way of specifying all hyperparameters for an experiment.** Pylearn2 accomplishes this using YAML with a few extensions. A brief YAML file can instantiate a complex experiment without exposing any implementation-specific detail. This makes it easier for researchers who do not use Pylearn2 to read the specification of a Pylearn2 experiment and reproduce it using their own software.

# Library overview

Pylearn2 consists of several components that can be combined to form complete learning algorithms. Most components do not actually execute any numerical code–they just provide symbolic expressions. This is possible because Pylearn2 is built on top of Theano . Theano provides a language for describing expressions independent of how they are actually implemented, so a single Pylearn2 class provides both CPU and GPU functionality. Another advantage of using symbolic representations as the main arguments to Pylearn2 methods is that it is possible to compute many functions of a symbolic expression that can not be computed from a numerical value alone. For example, it is possible to compute the derivative of a Theano expression, while it is not possible to compute the derivative of the process that generated a numerical value given only the value itself. This means that many interfaces are simpler–few expressions need to be passed between objects, because the recipient can create its own modifications of the expression it is passed, rather than needing an interface for each modified value it requires.

## Core components

The main way the Pylearn2 achieves flexibility and extensibility is decomposition into reusable parts. The three key components used to implement most features are the `Dataset`, `Model`, and `TrainingAlgorithm` classes. A `Dataset` provides the data to be trained on. A `Model` stores parameters and can generate Theano expressions that perform some useful task given input data (e.g., estimate the probability density at a point in space, infer a class label given input features, etc.). A `TrainingAlgorithm` adapts a `Model` to a particular `Dataset`. Generally each of these objects is in turn modular (`Dataset`s have modular preprocessing, many `Model` classes are organized into `Layer`s, `TrainingAlgorithms` can minimize a modular `Cost` and can have their behavior modified by various modular callbacks and a modular `TerminationCriterion`, etc.). This modularity means that if a researcher has an innovative idea to test out, and that idea only affects one component, the researcher can simply replace or subclass the component in question. The vast majority of the learning system can still be used with the new idea.

This modularity is in contrast to most other machine learning libraries, where the `Model` generally does most of the work. A scikit-learn model is generally accompanied by a `fit` method that is a complete training algorithm and that can't be applied to any other model. Some libraries are more modular but don't entirely divide the labor between models and training algorithms as sharply as Pylearn2 does. For example, in Torch  or DistBelief  the `Model`s are modular, but to train a layer, the layer needs to implement at the very least a backpropogation method for computing derivatives. In Pylearn2, the `Model` is only responsible for creating symbolic expressions, which the `TrainingAlgorithm` may or may not symbolically differentiate at a later time. (Individual Theano ops must still implement a `grad` method, but a comparatively small number of basic ops can be used to implement the comparatively large number of more complex models that appear in most machine learning libraries).

However, another aspect of Pylearn2's design philosophy is that no user should be forced to learn the entire framework. It's possible to just implement a `train_batch` method and have the `DefaultTrainingAlgorithm` do nothing but serve the `Model` batches of data from the `Dataset`, or to ignore `TrainingAlgorithm`s altogether and just pass a `Dataset` to the `Model`'s `train_all` method.

To facilitate code reuse, whenever possible, individual components that are shipped with the library aim to be as modular and orthogonal as possible, relying on other existing components – e.g., most `Model`s that become part of the library will defer their learning functionality to an existing `TrainingAlgorithm` and/or `Cost` unless sufficiently specialized as to be infeasible.

## The `Dataset` class

`Dataset`s are essentially interfaces between sources of data in arbitrary format and the in-memory array formats that Pylearn2 expects. All Pylearn2 `Dataset`s provide the same interface but can be implemented to use any back-end format. Currently, all Pylearn2 `Dataset`s just read data from disk, but in principle a `Dataset` could access live streaming data from the network or a peripheral device like a webcam.

`Dataset`s allow the data to be presented in many formats, regardless of how it is stored. For example, a minibatch of 64 different $32 \times 32$ pixel RGB images could be presented as a $64 \times 3072$ element design matrix, or it could be presented as a $64 \times 32 \times 32 \times 3$ tensor. The choice of which attribute to put on which axis can change to support different pieces of software too (for example, Theano convolution prefers batch size $\times$ channels $\times$ rows $\times$ columns, while cuda-convnet  (which is wrapped in Pylearn2) prefers channels $\times$ rows $\times$ columns $\times$ batch size). The data can also be presented in many different orders, allowing iteration in sequential or different types of random order. `Dataset`s can implement as many or as few iteration schemes as the implementor wants to–depending on the back end of the data, not all iteration schemes are efficient or even possible (for example, if the iterator needs to read from a network drive, random iteration may be very slow, and if the iterator reads live video from a webcam, there is no way for it to visit the future).

Most `Dataset`s used in the deep learning community can be represented as a design matrix stored in dense matrix format. For these datasets, implementing an appropriate `Dataset` object is very easy. The implementer only needs to subclass the `DenseDesignMatrix` class and implement a constructor that loads the desired data. If the data is already stored in NumPy  or pickle format, it is not even necessary to implement any new Python code to use the dataset.

Some datasets can be described as dense design matrices but are too big to fit into memory. Pylearn2 supports this use case via the `DenseDesignMatrixPyTables`. To use this class, the data must be stored in HDF5 format on disk.

Most `Dataset`s also support some kind of preprocessing that can modify the data after it has been loaded.

### Implemented `Dataset`s

Pylearn2 currently contains wrappers for several datasets. These include the datasets used for DARPA's unsupervised and transfer learning challenge , the dataset used for the NIPS 2011 workshops challenge in representation learning , the CIFAR-10 and CIFAR-100 datasets , the MNIST dataset , some of the MNIST variations datasets , the NORB dataset , the Street View House Numbers dataset , the Toronto Faces Database , some of the UCI repository datasets , and a dataset of 3D animations of fish . Additionally, there are many kinds of preprocessing, such as PCA , ZCA , various kinds of local contrast normalization , as well as helper functions to set up the entire preprocessing pipeline from some well-known successful and documented systems .

## The `Model` class

A `Model` is any object that stores parameters (for the purpose of Pylearn2, a "non-parametric" model is just one with a variable number of parameters). The basic `Model` class has very few interface elements. Subclasses of the `Model` class define richer interfaces. These interfaces define different quantities that the `Model` can compute. For example, the `MLP` class provides an `fprop` method that provides a symbolic expression for forward propagation in a multilayer perceptron. If the final layer of the MLP is a softmax layer representing distributions over classes $y$, and the `fprop` method is passed a Theano variable representing inputs $x$, the output will be a Theano variable representing $p(y \mid x)$.

The `Model` class is not required to know how to train itself, though many models do.

### Linear operators, spaces, and convolution

Linear operations are key parts of many machine learning models. The distinction between many important classes of machine learning models is often nothing more than what specific structure of linear transformation they use. For example, both MLPs and convolutional networks apply linear operators followed by a nonlinearity to transform inputs into outputs. In the MLP, the linear operation is multiplication by a dense matrix. In a convolutional network, the linear operation is discrete convolution with finite support. This operation can be viewed as matrix multiplication by a sparse matrix with several elements of the matrix constrained to be equal to each other. The point is that both use a linear transformation. Pylearn2's `LinearTransform` class provides a generic representation of linear operators. Pylearn2 functionality written using this class can thus be written once and then extended to do dense matrix multiplication, convolution, tiled convolution, local connections, etc. simply by providing different implementations of the linear operator. This idea grew out of James Bergstra's Theano-linear module which has since been incorporated into Pylearn2.

Different linear operator implementations require their inputs to be formatted in different ways. For example, convolution applied to an image requires a format that indicates the 2D position of each element of the input, while dense matrix multiplication just requires a linearized vector representaiton of the image. In Pylearn2, classes called `Space`s represent these different views of the same underlying data. Dense matrix multiplication acts on data that lives in a `VectorSpace`, while 2D convolution acts on data that lives in a `Conv2DSpace`. `Space`s generally know how to convert between each other, when possible. For example, an image in a `Conv2DSpace` can be flattened into a vector in a `VectorSpace`.

Several linear operators (and related convolutional network operations like spatial max pooling) in Pylearn2 are implemented as wrappers that add Theano semantics on top of the extremely fast cuda-convnet library , making Pylearn2 a very practical library to use for convolutional network research.

### Implemented models

Because the philosophy that Pylearn2 developers should write features when they are needed, and because most Pylearn2 developers so far have been deep learning researchers, Pylearn2 mostly contains deep learning models or models that are used as building blocks for deep architectures. This includes autoencoders , RBMs  including RBMs with Gaussian visible units , DBMs , MLPs , convolutional networks , and local coordinate coding . However, Pylearn2 is not restricted to deep learning functionality. We encourage submissions of other machine learning models. Often, LISA researchers work on problems whose scale exceeds that of the typical machine learning library user, so we occasionally implement features for simpler algorithms, such as SVMs  with reduced memory consumption or k-means  with fast multicore training.

Pylearn2 has implementations of several models that were developed at LISA, including denoising auto-encoders (DAEs) , contractive auto-encoders (CAEs)  including higher-order CAEs , spike-and-slab RBMs (ssRBMs)  including ssRBMs with pooling , reconstruction sampling autoencoders , and deep sparse rectifier nets . Pylearn2 also contains models that were developed not just at LISA but originally developed using Pylearn2, such as spike-and-slab sparse coding with parallel variational inference  and maxout units for neural nets .

## The `TrainingAlgorithm` class

The role of the `TrainingAlgorithm` class is to adjust the parameters stored in a `Model` in order to adapt the model to a given `Dataset`. The `TrainingAlgorithm` is also responsible for a few less important tasks, such as setting up the `Monitor` to record various values throughout training (to make learning curves, essentially). The `TrainingAlgorithm` is one of the very few Pylearn2 classes that actually performs numerical computation. It gathers Theano expressions assembled by the `Model` and other classes, synthesizes them into expressions for learning rules, compiles the learning rules into Theano functions that accomplish the learning, and executes the Theano expressions. In fact, Pylearn2 `TrainingAlgorithms` are not even required to use Theano at all. Some use a mixture of Theano and generic Python code; for example to perform line searches with the control logic done with basic Python loops and branching but the numerical computation done by Theano.

Most `TrainingAlgorithms` support constrained optimization by asking the `Model` to project the result of each learning update back into an allowed region. Many Pylearn2 models impose non-negativity constraints on parameters that represent, for example, the conditional variance of some random variable, and most Pylearn2 neural network layers support max norm constraints on the weights .

### The `Cost` class

Many training algorithms can be expressed as procedures for iteratively minimizing a cost function. This provides another opportunity for sharing code between algorithms. The `Cost` class represents a cost function independent of the algorithm used to minimize it, and each `TrainingAlgorithm` is free to use this representation or not, depending on what is most appropriate. A `Cost` is essentially just a class for generating a Theano expression describing the cost function, but it has a few extra pieces of functionality. For example, it can add monitoring channels that are relevant to the cost being minimized (for example, one popular cost is the negative log likelihood of class labels under a softmax model–this cost automatically adds a monitoring channel that tracks the misclassification rate of the classifier). One extremely important aspect of the `Cost` is that it has a `get_gradients` method. Unlike Theano's `grad` method, this method is not guaranteed to return accurate gradients. This allows many algorithms that use approximate gradients to be implemented using the same machinery as algorithms that use the exact gradient. For example, the persistent contrastive divergence  algorithm minimizes an intractable cost with intractable gradients–the log likelihood of a Boltzmann machine. The Pylearn2 `Cost` for persistent contrastive divergence returns `None` for the value of the cost function itself, to express that the cost function can't be computed. However, the standard `SGD` class is still able to perform stochastic gradient descent on the cost, because the `get_gradients` method for the cost returns a sampling-based approximation to the gradient. No special optimization class is needed to handle this seemingly exotic case.

Other costs implemented in the library include dropout , contrastive divergence , noise contrastive estimation , score matching , denoising score matching , softmax log likelihood for classification with MLPs, and Gaussian log likelihood for regression with MLPs. Many additional simpler costs can be added together with the `SumOfCosts` class to combine these primary costs with secondary costs to add regularization, such as weight decay or sparsity regularization.

### Implemented `TrainingAlgorithm`s

Currently, Pylearn2 contains three main `TrainingAlgorithm` classes. The `DefaultTrainingAlgorithm` does nothing but serve minibatches of data to the `Model`'s default minibatch learning rule. The `SGD` class does stochastic gradient descent on a `Cost`. This class supports extensions including Polyak averaging , momentum , and early stopping. The `BGD` class does batch gradient descent (in practice, large minibatches) aka the method of steepest descent . The `BGD` class is able to accumulate contributions to the gradient from several minibatches before making an update, thereby enabling it to use batches that are too large to fit in memory. Optional flags enable the `BGD` class to implement other similar algorithms such as nonlinear conjugate gradient descent .

# Development workflow and user community

Pylearn2 has many kinds of users and developers. One need not be a Pylearn2 developer to do research with Pylearn2. Pylearn2 is a valuable research tool even for people who do not need to develop any new algorithms. The wide array of reference implementations available in Pylearn2 make it useful for studying how existing algorithms behave under various conditions, or for obtaining baseline results on new tasks.

Researchers who wish to implement new algorithms with Pylearn2 do not necessarily need to become Pylearn2 developers either. It's common to develop experimental features privately in an offline repository. It's also perfectly fine to share Pylearn2 classes as part of a 3rd party repository rather than having them merged to the main Pylearn2 repository.

For those who do wish to contribute to Pylearn2, thank you! The process is designed to make sure the library is as stable as possible. Developers should first write to `email@example.com` to plan how to implement their feature. If the feature requires a change to existing APIs, it's important to follow the best practices guide[^1]. Once a plan is in place, developers should write the feature in their own fork of Pylearn2 on GitHub, then submit a pull request to the main repository. Our automated test suite will run on the pull request and indicate whether it is safe to merge. Pylearn2 developers will also review the pull request. When both the automatic tests and the reviewers are satisfied, one of us will merge the pull request. Be sure to write to `pylearn-dev` to find a reviewer for your pull request.

All kinds of pull requests are welcome–new features (provided that they have tests), config files for important results, bug fixes, and tests for existing features.

# Conclusion

This article has described the Pylearn2 library, including its history, design philosophy and goals, basic architecture, and developer workflow. We hope you find Pylearn2 useful in your research and welcome your potential contributions to it.

[^1]: <http://deeplearning.net/software/pylearn2/api_change.html>


abstract: In 1686 in his *Discours de métaphysique,* Leibniz points out that if an arbitrarily complex theory is permitted then the notion of "theory" becomes vacuous because there is always a theory. This idea is developed in the modern theory of algorithmic information, which deals with the size of computer programs and provides a new view of Gödel's work on incompleteness and Turing's work on uncomputability. Of particular interest is the halting probability $\Omega$, whose bits are irreducible, i.e., maximally unknowable mathematical facts. More generally, these ideas constitute a kind of "digital philosophy" related to recent attempts of Edward Fredkin, Stephen Wolfram and others to view the world as a giant computer. There are also connections with recent "digital physics" speculations that the universe might actually be discrete, not continuous. This *système du monde* is presented as a coherent whole in my book *Meta Math!,* which will be published this fall.
author: **Gregory Chaitin**[^1]
title: **Epistemology as Information Theory:
  From Leibniz to $\Omega$[^2]**

# Introduction

I am happy to be here with you enjoying the delicate Scandinavian summer; if we were a little farther north there wouldn't be any darkness at all. And I am especially delighted to be here delivering the Alan Turing Lecture. Turing's famous 1936 paper is an intellectual milestone that seems larger and more important with every passing year.[^3]

People are not merely content to enjoy the beautiful summers in the far north, they also want and need **to understand**, and so they create myths. In this part of the world those myths involve Thor and Odin and the other Norse gods. In this talk, I'm going to present another myth, what the French call a *système du monde,* a system of the world, a speculative metaphysics based on information and the computer.[^4]

The previous century had logical positivism and all that emphasis on the philosophy of language, and completely shunned speculative metaphysics, but a number of us think that it is time to start again. There is an emerging digital philosophy and digital physics, a new metaphysics associated with names like Edward Fredkin and Stephen Wolfram and a handful of like-minded individuals, among whom I include myself. As far as I know the terms "digital philosophy" and "digital physics" were actually invented by Fredkin, and he has a large website with his papers and a draft of a book about this. Stephen Wolfram attracted a great deal of attention to the movement and stirred up quite a bit of controversy with his very large and idiosyncratic book on *A New Kind of Science.*

And I have my own book on the subject, in which I've attempted to wrap everything I know and care about into a single package. It's a small book, and amazingly enough it's going to be published by a major New York publisher a few months from now. This talk will be an overview of my book, which presents my own personal version of "digital philosophy," since each of us who works in this area has a different vision of this tentative, emerging world view. My book is called *Meta Math!,* which may not seem like a serious title, but it's actually a book intended for my professional colleagues as well as for the general public, the high-level, intellectual, thinking public.

"Digital philosophy" is actually a neo-Pythagorean vision of the world, it's just a new version of that. According to Pythagoras, all is number — and by number he means the positive integers, 1, 2, 3, … — and God is a mathematician. "Digital philosophy" updates this as follows: Now everything is made out of 0/1 bits, everything is digital software, and God is a computer programmer, not a mathematician! It will be interesting to see how well this vision of the world succeeds, and just how much of our experience and theorizing can be included or shoe-horned within this new viewpoint.[^5]

Let me return now to Turing's famous 1936 paper. This paper is usually remembered for inventing the programmable digital computer via a mathematical model, the Turing machine, and for discovering the extremely fundamental halting problem. Actually Turing's paper is called "On computable numbers, with an application to the *Entscheidungsproblem,*" and by computable numbers Turing means "real" numbers, numbers like $e$ or $\pi = 3.1415926\ldots$ that are measured with infinite precision, and that can be computed with arbitrarily high precision, digit by digit without ever stopping, on a computer.

Why do I think that Turing's paper "On computable numbers" is so important? Well, in my opinion it's a paper on epistemology, because we only understand something if we can program it, as I will explain in more detail later. And it's a paper on physics, because what we can actually compute depends on the laws of physics in our particular universe and distinguishes it from other possible universes. And it's a paper on ontology, because it shows that some real numbers are **uncomputable**, which I shall argue calls into question their very existence, their mathematical and physical existence.[^6]

To show how strange uncomputable real numbers can be, let me give a particularly illuminating example of one, which actually preceded Turing's 1936 paper. It's a very strange number that was invented in a 1927 paper by the French mathematician Emile Borel. Borel's number is sort of an anticipation, a partial anticipation, of Turing's 1936 paper, but that's only something that one can realize in retrospect. Borel presages Turing, which does not in any way lessen Turing's important contribution that so dramatically and sharply clarified all these vague ideas.[^7]

Borel was interested in "constructive" mathematics, in what you can actually compute we would say nowadays. And he came up with an extremely strange non-constructive real number. You list all possible yes/no questions in French in an immense, an infinite list of all possibilities. This will be what mathematicians call a denumerable or a countable infinity of questions, because it can be put into a one-to-one correspondence with the list of positive integers 1, 2, 3, … In other words, there will be a first question, a second question, a third question, and in general an $N$th question.

You can imagine all the possible questions to be ordered by size, and within questions of the same size, in alphabetical order. More precisely, you consider all possible strings, all possible finite sequences of symbols in the French alphabet, including the blank so that you get words, and the period so that you have sentences. And you imagine filtering out all the garbage and being left only with grammatical yes/no questions in French. Later I will tell you in more detail how to actually do this. Anyway, for now **imagine** doing this, and so there will be a first question, a second question, an $N$th question.

And the $N$th digit or the $N$th bit after the decimal point of Borel's number answers the $N$th question: It will be a 0 if the answer is no, and it'll be a 1 if the answer is yes. So the binary expansion of Borel's number contains the answer to every possible yes/no question! It's like having an oracle, a Delphic oracle that will answer every yes/no question!

How is this possible?! Well, according to Borel, it isn't really possible, this can't be, it's totally unbelievable. This number is only a mathematical fantasy, it's not for real, it cannot claim a legitimate place in our ontology. Later I'll show you a modern version of Borel's number, my halting probability $\Omega$. And I'll tell you why some contemporary physicists, real physicists, not mavericks, are moving in the direction of digital physics.

\[Actually, to make Borel's number as real as possible, you have to avoid the problem of filtering out all the yes/no questions. And you have to use decimal digits, you can't use binary digits. You number all the possible finite strings of French symbols including blanks and periods, which is quite easy to do using a computer. Then the $N$th digit of Borel's number is 0 if the $N$th string of characters in French is ungrammatical and not proper French, it's 1 if it's grammatical, but not a yes/no question, it's 2 if it's a yes/no question that cannot be answered (e.g., "Is the answer to this question 'no'?"), it's 3 if the answer is no, and it's 4 if the answer is yes.\]

*Geometrically* a real number is the most straightforward thing in the world, it's just a point on a line. That's quite natural and intuitive. But *arithmetically,* that's another matter. The situation is quite different. From an arithmetical point of view reals are extremely problematical, they are fraught with difficulties!

Before discussing my $\Omega$ number, I want to return to the fundamental question of what does it mean to understand. How do we explain or comprehend something? What is a theory? How can we tell whether or not it's a successful theory? How can we measure how successful it is? Well, using the ideas of information and computation, that's not difficult to do, and the central idea can even be traced back to Leibniz's 1686 *Discours de métaphysique.*

# Computer Epistemology: What is a mathematical or scientific theory? How can we judge whether it works or not?

In Sections V and VI of his *Discourse on Metaphysics,* Leibniz asserts that God simultaneously maximizes the variety, diversity and richness of the world, and minimizes the conceptual complexity of the set of ideas that determine the world. And he points out that for any finite set of points there is always a mathematical equation that goes through them, in other words, a law that determines their positions. But if the points are chosen at random, that equation will be extremely complex.

This theme is taken up again in 1932 by Hermann Weyl in his book *The Open World* consisting of three lectures he gave at Yale University on the metaphysics of modern science. Weyl formulates Leibniz's crucial idea in the following extremely dramatic fashion: If one permits arbitrarily complex laws, then the concept of law becomes vacuous, because there is always a law! Then Weyl asks, how can we make more precise the distinction between mathematical simplicity and mathematical complexity? It seems to be very hard to do that. How can we measure this important parameter, without which it is impossible to distinguish between a successful theory and one that is completely unsuccessful?

This problem is taken up and I think satisfactorily resolved in the new mathematical theory I call *algorithmic information theory.* The epistemological model that is central to this theory is that a scientific or mathematical theory is a computer program for calculating the facts, and the smaller the program, the better. The complexity of your theory, of your law, is measured in bits of software:

program (bit string) $\longrightarrow$ **Computer** $\longrightarrow$ output (bit string)
theory $\longrightarrow$ **Computer** $\longrightarrow$ mathematical or scientific facts
Understanding is compression!

Now Leibniz's crucial observation can be formulated much more precisely. For any finite set of scientific or mathematical facts, there is always a theory that is exactly as complicated, exactly the same size in bits, as the facts themselves. (It just directly outputs them "as is," without doing any computation.) But that doesn't count, that doesn't enable us to distinguish between what can be comprehended and what cannot, because there is always a theory that is as complicated as what it explains. A theory, an explanation, is only successful to the extent to which it compresses the number of bits in the facts into a much smaller number of bits of theory. Understanding is compression, comprehension is compression! That's how we can tell the difference between real theories and *ad hoc* theories.[^8]

What can we do with this idea that an explanation has to be simpler than what it explains? Well, the most important application of these ideas that I have been able to find is in metamathematics, it's in discussing what mathematics can or cannot achieve. You simultaneously get an information-theoretic, computational perspective on Gödel's famous 1931 incompleteness theorem, and on Turing's famous 1936 halting problem. How?[^9]

Here's how! These are my two favorite information-theoretic incompleteness results:

-   You need an $N$-bit theory in order to be able to prove that a specific $N$-bit program is "elegant."

-   You need an $N$-bit theory in order to be able to determine $N$ bits of the numerical value, of the base-two binary expansion, of the halting probability $\Omega$.

Let me explain.

What is an elegant program? It's a program with the property that no program written in the same programming language that produces the same output is smaller than it is. In other words, an elegant program is the most concise, the simplest, the best theory for its output. And there are infinitely many such programs, they can be arbitrarily big, because for any computational task there has to be at least one elegant program. (There may be several if there are ties, if there are several programs for the same output that have exactly the minimum possible number of bits.)

And what is the halting probability $\Omega$? Well, it's defined to be the probability that a computer program generated at random, by choosing each of its bits using an independent toss of a fair coin, will eventually halt. Turing is interested in whether or not individual programs halt. I am interested in trying to prove what are the bits, what is the numerical value, of the halting probability $\Omega$. By the way, the value of $\Omega$ depends on your particular choice of programming language, which I don't have time to discuss now. $\Omega$ is also equal to the result of summing 1/2 raised to powers which are the size in bits of every program that halts. In other words, each $K$-bit program that halts contributes $1/2^K$ to $\Omega$.

And what precisely do I mean by an $N$-bit mathematical theory? Well, I'm thinking of formal axiomatic theories, which are formulated using symbolic logic, not in any natural, human language. In such theories there are always a finite number of axioms and there are explicit rules for mechanically deducing consequences of the axioms, which are called theorems. An $N$-bit theory is one for which there is an $N$-bit program for systematically running through the tree of all possible proofs deducing all the consequences of the axioms, which are all the theorems in your formal theory. This is slow work, but in principle it can be done mechanically, that's what counts. David Hilbert believed that there had to be a single formal axiomatic theory for all of mathematics; that's just another way of stating that math is static and perfect and provides absolute truth.

Not only is this impossible, not only is Hilbert's dream impossible to achieve, but there are in fact an infinity of irreducible mathematical truths, mathematical truths for which essentially the only way to prove them is to add them as new axioms. My first example of such truths was determining elegant programs, and an even better example is provided by the bits of $\Omega$. The bits of $\Omega$ are mathematical facts that are true for no reason (no reason simpler than themselves), and thus violate Leibniz's principle of sufficient reason, which states that if anything is true it has to be true for a reason.

In math the reason that something is true is called its proof. Why are the bits of $\Omega$ true for no reason, why can't you prove what their values are? Because, as Leibniz himself points out in Sections 33 to 35 of *The Monadology,* the essence of the notion of proof is that you prove a complicated assertion by analyzing it, by breaking it down until you reduce its truth to the truth of assertions that are so simple that they no longer require any proof (self-evident axioms). But if you cannot deduce the truth of something from any principle simpler than itself, then proofs become useless, because **anything** can be proven from principles that are equally complicated, e.g., by directly adding it as a new axiom without any proof. And this is exactly what happens with the bits of $\Omega$.

**In other words, the normal, Hilbertian view of math is that all of mathematical truth, an infinite number of truths, can be compressed into a finite number of axioms. But there are an infinity of mathematical truths that cannot be compressed at all, not one bit!**

This is an amazing result, and I think that it has to have profound philosophical and practical implications. Let me try to tell you why.

On the one hand, it suggests that pure math is more like biology than it is like physics. In biology we deal with very complicated organisms and mechanisms, but in physics it is normally assumed that there has to be a theory of everything, a simple set of equations that would fit on a T-shirt and in principle explains the world, at least the physical world. But we have seen that the world of mathematical ideas has infinite complexity, it cannot be explained with any theory having a finite number of bits, which from a sufficiently abstract point of view seems much more like biology, the domain of the complex, than like physics, where simple equations reign supreme.

On the other hand, this amazing result suggests that even though math and physics are different, they may not be as different as most people think! I mean this in the following sense: In math you organize your computational experience, your lab is the computer, and in physics you organize physical experience and have real labs. But in both cases an explanation has to be simpler than what it explains, and in both cases there are sets of facts that cannot be explained, that are irreducible. Why? Well, in quantum physics it is assumed that there are phenomena that when measured are equally likely to give either of two answers (e.g., spin up, spin down) and that are inherently unpredictable and irreducible. And in pure math we have a similar example, which is provided by the individual bits in the binary expansion of the numerical value of the halting probability $\Omega$.

This suggests to me a quasi-empirical view of math, in which one is more willing to add new axioms that are not at all self-evident but that are justified pragmatically, i.e., by their fruitful consequences, just like a physicist would. I have taken the term quasi-empirical from Lakatos. The collection of essays *New Directions in the Philosophy of Mathematics* edited by Tymoczko in my opinion pushes strongly in the direction of a quasi-empirical view of math, and it contains an essay by Lakatos proposing the term "quasi-empirical," as well as essays of my own and by a number of other people. Many of them may disagree with me, and I'm sure do, but I repeat, in my opinion all of these essays justify a quasi-empirical view of math, what I mean by quasi-empirical, which is somewhat different from what Lakatos originally meant, but is in quite the same spirit, I think.

In a two-volume work full of important mathematical examples, Borwein, Bailey and Girgensohn have argued that experimental mathematics is an extremely valuable research paradigm that should be openly acknowledged and indeed vigorously embraced. They do not go so far as to suggest that one should add new axioms whenever they are helpful, without bothering with proofs, but they are certainly going in that direction and nod approvingly at my attempts to provide some theoretical justification for their entire enterprise by arguing that math and physics are not that different.

In fact, since I began to espouse these heretical views in the early 1970's, largely to deaf ears, there have actually been several examples of such new pragmatically justified, non-self-evident axioms:

-   the $\mathbf{P \neq NP}$ hypothesis regarding the time complexity of computations,

-   the axiom of projective determinacy in set theory, and

-   increasing reliance on diverse unproved versions of the Riemann hypothesis regarding the distribution of the primes.

So people don't need to have theoretical justification; they just do whatever is needed to get the job done…

The only problem with this computational and information-theoretic epistemology that I've just outlined to you is that it's based on the computer, and there are uncomputable reals. So what do we do with contemporary physics which is full of partial differential equations and field theories, all of which are formulated in terms of real numbers, **most of which are in fact uncomputable**, as I'll now show. Well, it would be good to get rid of all that and convert to a *digital physics.* Might this in fact be possible?! I'll discuss that too.

# Computer Ontology: How real are real numbers? What is the world made of?

How did Turing prove that there are uncomputable reals in 1936? He did it like this. Recall that the possible texts in French are a countable or denumerable infinity and can be placed in an infinite list in which there is a first one, a second one, etc. Now let's do the same thing with all the possible computer programs (first you have to choose your programming language). So there is a first program, a second program, etc. Every computable real can be calculated digit by digit by some program in this list of all possible programs. Write the numerical value of that real next to the programs that calculate it, and cross off the list all the programs that do not calculate an individual computable real. We have converted a list of programs into a list of computable reals, and no computable real is missing.

Next discard the integer parts of all these computable reals, and just keep the decimal expansions. Then put together a new real number by changing every digit on the diagonal of this list (this is called Cantor's diagonal method; it comes from set theory). So your new number's first digit differs from the first digit of the first computable real, its second digit differs from the second digit of the second computable real, its third digit differs from the third digit of the third computable real, and so forth and so on. So it can't be in the list of all computable reals and it has to be uncomputable. And that's Turing's uncomputable real number![^10]

Actually, there is a much easier way to see that there are uncomputable reals by using ideas that go back to Emile Borel (again!). Technically, the argument that I'll now present uses what mathematicians call *measure theory,* which deals with probabilities. So let's just look at all the real numbers between 0 and 1. These correspond to points on a line, a line exactly one unit in length, whose leftmost point is the number 0 and whose rightmost point is the number 1. The total length of this line segment is of course exactly one unit. But I will now show you that all the computable reals in this line segment can be covered using intervals whose total length can be made as small as desired. In technical terms, the computable reals in the interval from 0 to 1 are a set of measure zero, they have zero probability.

How do you cover all the computable reals? Well, remember that list of all the computable reals that we just diagonalized over to get Turing's uncomputable real? This time let's cover the first computable real with an interval of size $\epsilon/2$, let's cover the second computable real with an interval of size $\epsilon/4$, and in general we'll cover the $N$th computable real with an interval of size $\epsilon/2^N$. The total length of all these intervals (which can conceivably overlap or fall partially outside the unit interval from 0 to 1), is exactly equal to $\epsilon$, which can be made as small as we wish! In other words, there are arbitrarily small coverings, and the computable reals are therefore a set of measure zero, they have zero probability, they constitute an infinitesimal fraction of all the reals between 0 and 1. So if you pick a real at random between 0 and 1, with a uniform distribution of probability, it is infinitely unlikely, though possible, that you will get a computable real!

What disturbing news! Uncomputable reals are not the exception, they are the majority! How strange!

In fact, the situation is even worse than that. As Emile Borel points out on page 21 of his final book, *Les nombres inaccessibles* (1952), without making any reference to Turing, most individual reals are not even uniquely specifiable, they cannot even be named or pointed out, no matter how non-constructively, because of the limitations of human languages, which permit only a countable infinity of possible texts. The individually accessible or nameable reals are also a set of measure zero. Most reals are un-nameable, with probability one! I rediscovered this result of Borel's on my own in a slightly different context, in which things can be done a little more rigorously, which is when one is dealing with a formal axiomatic theory or an **artificial** formal language instead of a natural human language. That's how I present this idea in *Meta Math!*.

So if most individual reals will forever escape us, why should we believe in them?! Well, you will say, because they have a pretty structure and are a nice theory, a nice game to play, with which I certainly agree, and also because they have important practical applications, they are needed in physics. Well, perhaps not! Perhaps physics can give up infinite precision reals! How? Why should physicists want to do that?

Because it turns out that there are actually many reasons for being skeptical about the reals, in classical physics, in quantum physics, and particularly in more speculative contemporary efforts to cobble together a theory of black holes and quantum gravity.

First of all, as my late colleague the physicist Rolf Landauer used to remind me, no physical measurement has ever achieved more than a small number of digits of precision, not more than, say, 15 or 20 digits at most, and such high-precision experiments are rare masterpieces of the experimenter's art and not at all easy to achieve.

This is only a practical limitation in classical physics. But in quantum physics it is a consequence of the Heisenberg uncertainty principle and wave-particle duality (de Broglie). According to quantum theory, the more accurately you try to measure something, the smaller the length scales you are trying to explore, the higher the energy you need (the formula describing this involves Planck's constant). That's why it is getting more and more expensive to build particle accelerators like the one at CERN and at Fermilab, and governments are running out of money to fund high-energy physics, leading to a paucity of new experimental data to inspire theoreticians.

Hopefully new physics will eventually emerge from astronomical observations of bizarre new astrophysical phenomena, since we have run out of money here on earth! In fact, currently some of the most interesting physical speculations involve the thermodynamics of black holes, massive concentrations of matter that seem to be lurking at the hearts of most galaxies. Work by Stephen Hawking and Jacob Bekenstein on the thermodynamics of black holes suggests that any physical system can contain only a finite amount of information, a finite number of bits whose possible maximum is determined by what is called the Bekenstein bound. Strangely enough, this bound on the number of bits grows as the surface area of the physical system, not as its volume, leading to the so-called "holographic" principle asserting that in some sense space is actually two-dimensional even though it appears to have three dimensions!

So perhaps continuity is an illusion, perhaps everything is really discrete. There is another argument against the continuum if you go down to what is called the Planck scale. At distances that extremely short our current physics breaks down because spontaneous fluctuations in the quantum vacuum should produce mini-black holes that completely tear spacetime apart. And that is not at all what we see happening around us. So perhaps distances that small **do not exist**.

Inspired by ideas like this, in addition to *a priori* metaphysical biases in favor of discreteness, a number of contemporary physicists have proposed building the world out of discrete information, out of bits. Some names that come to mind in this connection are John Wheeler, Anton Zeilinger, Gerard 't Hooft, Lee Smolin, Seth Lloyd, Paola Zizzi, Jarmo Mäkelä and Ted Jacobson, who are real physicists. There is also more speculative work by a small cadre of cellular automata and computer enthusiasts including Edward Fredkin and Stephen Wolfram, whom I already mentioned, as well as Tommaso Toffoli, Norman Margolus, and others.

And there is also an increasing body of highly successful work on quantum computation and quantum information that is not at all speculative, it is just a fundamental reworking of standard 1920's quantum mechanics. Whether or not quantum computers ever become practical, the workers in this highly popular field have clearly established that it is illuminating to study sub-atomic quantum systems in terms of how they process qubits of quantum information and how they perform computation with these qubits. These notions have shed completely new light on the behavior of quantum mechanical systems.

Furthermore, when dealing with complex systems such as those that occur in biology, thinking about information processing is also crucial. As I believe Seth Lloyd said, the most important thing in understanding a complex system is to determine how it represents information and how it processes that information, i.e., what kinds of computations are performed.

And how about the entire universe, can it be considered to be a computer? Yes, it certainly can, it is constantly computing its future state from its current state, it's constantly computing its own time-evolution! And as I believe Tom Toffoli pointed out, actual computers like your PC just hitch a ride on this universal computation!

So perhaps we are not doing violence to Nature by attempting to force her into a digital, computational framework. Perhaps she has been flirting with us, giving us hints all along, that she is really discrete, not continuous, hints that we choose not to hear, because we are so much in love and don't want her to change!

For more on this kind of new physics, see the books by Smolin and von Baeyer in the bibliography. Several more technical papers on this subject are also included there.

# Conclusion

Let me now wrap this up and try to give you a present to take home, more precisely, a piece of homework. In extremely abstract terms, I would say that the problem is, as was emphasized by Ernst Mayr in his book *This is Biology,* that the current philosophy of science deals more with physics and mathematics than it does with biology. But let me try to put this in more concrete terms and connect it with the spine, with the central thread, of the ideas in this talk.

To put it bluntly, a closed, static, eternal fixed view of math can no longer be sustained. As I try to illustrate with examples in my *Meta Math!* book, math actually advances by inventing new concepts, by completely changing the viewpoint. Here I emphasized new axioms, increased complexity, more information, but what really counts are new ideas, new concepts, new viewpoints. And that leads me to the crucial question, crucial for a proper open, dynamic, time-dependent view of mathematics,

**"Where do new mathematical ideas come from?"**

I repeat, math does not advance by mindlessly and mechanically grinding away deducing all the consequences of a fixed set of concepts and axioms, not at all! It advances with new concepts, new definitions, new perspectives, through revolutionary change, paradigm shifts, not just by hard work.

In fact, I believe that this is actually the central question in biology as well as in mathematics, it's the mystery of creation, of creativity:

**"Where do new mathematical and biological ideas come from?"**
**"How do they emerge?"**

Normally one equates a new biological idea with a new species, but in fact every time a child is born, that's actually a new idea incarnating; it's reinventing the notion of "human being," which changes constantly.

I have no idea how to answer this extremely important question; I wish I could. Maybe **you** will be able to do it. Just try! You might have to keep it cooking on a back burner while concentrating on other things, but don't give up! All it takes is a new idea! Somebody has to come up with it. Why not you?[^11]

# Appendix: Leibniz and the Law

I am indebted to Professor Ugo Pagallo for explaining to me that Leibniz, whose ideas and their elaboration were the subject of my talk, is regarded as just as important in the field of law as he is in the fields of mathematics and philosophy.

The theme of my lecture was that if a law is arbitrarily complicated, then it is not a law; this idea was traced via Hermann Weyl back to Leibniz. In mathematics it leads to my $\Omega$ number and the surprising discovery of completely lawless regions of mathematics, areas in which there is absolutely no structure or pattern or way to understand what is happening.

The principle that an arbitrarily complicated law is not a law can also be interpreted with reference to the legal system. It is not a coincidence that the words "law" and "proof" and "evidence" are used in jurisprudence as well as in science and mathematics. In other words, the rule of law is equivalent to the rule of reason, but if a law is sufficiently complicated, then it can in fact be completely arbitrary and incomprehensible.

# Acknowledgements

I wish to thank Gordana Dodig-Crnkovic for organizing E-CAP'05 and for inviting me to present the Turing lecture at E-CAP'05; also for stimulating discussions reflected in those footnotes that are marked with GDC. The remarks on biology are the product of a week spent in residence at Rockefeller University in Manhattan, June 2005; I thank Albert Libchaber for inviting me to give a series of lectures there to physicists and biologists. The appendix is the result of lectures to philosophy of law students April 2005 at the Universities of Padua, Bologna and Turin; I thank Ugo Pagallo for arranging this. Thanks too to Paola Zizzi for help with the physics references.

# References

-   Edward Fredkin, http://www.digitalphilosophy.org.

-   Stephen Wolfram, *A New Kind of Science,* Wolfram Media, 2002.

-   Gregory Chaitin, *Meta Math!,* Pantheon, 2005.

-   G. W. Leibniz, *Discourse on Metaphysics, Principles of Nature and Grace, The Monadology,* 1686, 1714, 1714.

-   Hermann Weyl, *The Open World,* Yale University Press, 1932.

-   Thomas Tymoczko, *New Directions in the Philosophy of Mathematics,* Princeton University Press, 1998.

-   Jonathan Borwein, David Bailey, Roland Girgensohn, *Mathematics by Experiment, Experimentation in Mathematics,* A. K. Peters, 2003, 2004.

-   Rebecca Goldstein, *Incompleteness,* Norton, 2005.

-   B. Jack Copeland, *The Essential Turing,* Oxford University Press, 2004.

-   Vladimir Tasic, *Mathematics and the Roots of Postmodern Thought,* Oxford University Press, 2001.

-   Emile Borel, *Les nombres inaccessibles,* Gauthier-Villars, 1952.

-   Lee Smolin, *Three Roads to Quantum Gravity,* Basic Books, 2001.

-   Hans Christian von Baeyer, *Information,* Harvard University Press, 2004.

-   Ernst Mayr, *This is Biology,* Harvard University Press, 1998.

-   J. Wheeler, (the "It from bit" proposal), *Sakharov Memorial Lectures on Physics,* vol. 2, Nova Science, 1992.

-   A. Zeilinger, (the principle of quantization of information), *Found. Phys.* **29**:631–643 (1999).

-   G. 't Hooft, "The holographic principle," http://arxiv.org/hep-th/0003004.

-   S. Lloyd, "The computational universe," http://arxiv.org/quant-ph/0501135.

-   P. Zizzi, "A minimal model for quantum gravity," http://arxiv.org/gr-qc/0409069.

-   J. Mäkelä, "Accelerating observers, area and entropy," http://arxiv.org/gr-qc/0506087.

-   T. Jacobson, "Thermodynamics of spacetime," http://arxiv.org/gr-qc/9504004.

[^1]: IBM T. J. Watson Research Center, P. O. Box 218, Yorktown Heights, NY 10598, U.S.A., *firstname.lastname@example.com.*

[^2]: Alan Turing Lecture on Computing and Philosophy, E-CAP'05, European Computing and Philosophy Conference, Mälardalen University, Västerås, Sweden, June 2005.

[^3]: For Turing's original paper, with commentary, see Copeland's *The Essential Turing.*

[^4]: One reader's reaction (GDC): "Grand unified theories may be like myths, but surely there is a difference between scientific theory and any other narrative?" I would argue that a scientific narrative is more successful than the Norse myths because it explains what it explains more precisely and without having to postulate new gods all the time, i.e., it's a better "compression" (which will be my main point in this lecture; that's how you measure how successful a theory is).

[^5]: Of course, a system of the world can only work by omitting everything that doesn't fit within its vision. The question is how much will fail to fit, and conversely, how many things will this vision be able to help us to understand. Remember, if one is wearing rose colored glasses, everything seems pink. And as Picasso said, theories are lies that help us to see the truth. No theory is perfect, and it will be interesting to see how far this digital vision of the world will be able to go.

[^6]: You might exclaim (GDC), "You can't be saying that before Turing and the computer no one understood anything; that can't be right!" My response to this is that before Turing (and my theory) people could understand things, but **they couldn't measure how well** they understood them. Now you can measure that, in terms of the degree of compression that is achieved. I will explain this later at the beginning of the section on computer epistemology. Furthermore, programming something forces you to understand it better, it forces you to really understand it, since you are explaining it **to a machine**. That's sort of what happens when a student or a small child asks you what at first you take to be a stupid question, and then you realize that this question has in fact done you the favor of forcing you to formulate your ideas more clearly and perhaps even question some of your tacit assumptions.

[^7]: I learnt of Borel's number by reading Tasic's *Mathematics and the Roots of Postmodern Thought,* which also deals with many of the issues discussed here.

[^8]: By the way, Leibniz also mentions complexity in Section 7 of his *Principles of Nature and Grace,* where he asks the amazing question, "Why is there something rather than nothing? For nothing is simpler and easier than something."

[^9]: For an insightful treatment of Gödel as a philosopher, see Rebecca Goldstein's *Incompleteness.*

[^10]: *Technical Note:* Because of **synonyms** like $.345999\ldots = .346000\ldots$ you should avoid having any 0 or 9 digits in Turing's number.

[^11]: I'm not denying the importance of Darwin's theory of evolution. But I want much more than that, I want a profound, extremely general mathematical theory that captures the essence of what life is and why it evolves. I want a theory that gets to the heart of the matter. And I suspect that any such theory will necessarily have to shed new light on mathematical creativity as well. Conversely, a deep theory of mathematical creation might also cover biological creativity.

    A reaction from Gordana Dodig-Crnkovic: "Regarding Darwin and Neo-Darwinism I agree with you — it is a very good idea to go beyond. In my view there is nothing more beautiful and convincing than a good mathematical theory. And I do believe that it must be possible to express those thoughts in a much more general way… I believe that it is a very crucial thing to try to formulate life in terms of computation. Not to say life is nothing more than a computation. But just to explore how far one can go with that idea. Computation seems to me a very powerful tool to illuminate many things about the material world and the material ground for mental phenomena (including creativity)… Or would you suggest that creativity is given by God's will? That it is the very basic axiom? Isn't it possible to relate to pure chance? Chance and selection? Wouldn't it be a good idea to assume two principles: law and chance, where both are needed to reconstruct the universe in computational terms? (like chaos and cosmos?)"