abstract: We introduce the concept of dynamically growing a neural network during training. In particular, an untrainable deep network starts as a trainable shallow network and newly added layers are slowly, organically added during training, thereby increasing the network's depth. This is accomplished by a new layer, which we call DropIn. The DropIn layer starts by passing the output from a previous layer (effectively skipping over the newly added layers), then increasingly including units from the new layers for both feedforward and backpropagation. We show that deep networks, which are untrainable with conventional methods, will converge with DropIn layers interspersed in the architecture. In addition, we demonstrate that DropIn provides regularization during training in an analogous way as dropout. Experiments are described with the MNIST dataset and various expanded LeNet architectures, CIFAR-10 dataset with its architecture expanded from 3 to 11 layers, and on the ImageNet dataset with the AlexNet architecture expanded to 13 layers and the VGG 16-layer architecture.
author: Leslie N. Smith
Naval Research Laboratory
`firstname.lastname@example.com`
; Emily M. Hand
University of Maryland
`firstname.lastname@example.com`; Timothy Doster
Naval Research Laboratory
`firstname.lastname@example.com`
bibliography: dropin.bib
title: Gradual DropIn of Layers to Train Very Deep Neural Networks

# Introduction

Over the past few years, state-of-the-art results for image recognition , object detection , face recognition , speech recognition , machine translation , image caption generation , driverless car technology , and other applications have required increasingly deeper neural networks.

Network depth refers to the number of layers in the architecture. It is well known that adding layers to neural networks makes them more expressive . Each year, the Imagenet Challenge is held in which teams are expected, given an image, to detect, localize, or recognize an object in the image. Deep convolutional neural networks (CNNs) have dominated the competition since Krizhevsky won in 2012 , and each year since, the winner of the competition used a deeper network than the previous year's winner .

However, training a very deep network is a difficult and open research problem . It is difficult to train very deep networks because the error norm during backpropagation can grow or vanish exponentially. In addition, very large training datasets are necessary when the network has millions of weights.

Here we suggest a dynamic architecture that grows during the training process and allows for the training of very deep networks. We illustrate this with our DropIn layer, where new layers are skipped at the start of the training, as though they were not present. This allows the weights of the included layers to start converging. Over a number of iterations the DropIn layer increasingly includes activations from the inserted layers, which gradually trains the weights in theses added layers.

DropIn follows the philosophy embedded within curriculum learning . With curriculum learning one starts with an easier problem and incrementally increases the difficulty. Here too, one starts training a shallow architecture and after convergence begins, DropIn incrementally modifies the architecture to slowly include units from the new layers.

In addition, DropIn can be used in a mode analogous to dropout for the regularization of a deep neural network during training. Instead of setting random activations to zero, as is done in dropout, DropIn sets these activations to the activations from a previous layer. We demonstrate that the "noise" from mixing the activations from previous layers provides regularization during training. In addition, both DropIn and dropout can be viewed as training a large collection of networks with varied architectures and extensive weight sharing.

The contributions of this paper are:

# Related work

Methods for training very deep networks have centered on initialization of the network weights or developing new architectures and DropIn is in the latter category.

## Initialization of network weights

Sutskever investigate the difficulty in training deep networks and conclude that both proper initialization and momentum are necessary. Glorot and Bengio recommend an initialization method called *normalized initialization* to allow the training of deep networks. He recently improved upon the "normalized initialization" method by changing the distribution to take into account ReLU layers.

Hinton proposed first training layer by layer in an unsupervised fashion so that a transformed version of the input could be realized. Erhan later characterized the mathematics of the unsupervised pre-training and offered an explanation for its success.

Sussillo and Abbott suggest an initialization scheme called *Random Walk Initialization* based on scaling the initial random matrices correctly. By multiplying the error gradient by a correctly scaled random matrix at each layer, an unbiased random walk is formed. This is one of only a few papers that show the results of experiments with networks consisting of hundreds of layers.

## Developing new architecture

Raiko, introduce the concept of skip connections by adding a linear transformation to the usual non-linear transformation of the input to a unit. Skip connections separate the linear and non-linear portions of the activations and allow the linear part to "skip" to higher layers. This is similar to DropIn in some ways, but the purpose of DropIn differs from that of skip connections, and DropIn does not need to learn any parameters.

Romero suggest training a thin, deep student network (called a *fitnet*) from a larger but shallower teacher network. The authors accomplish this by utilizing the output of the teacher's hidden layers as a hint for the student's hidden layers.

Srivastava propose a new architecture, which they named *Highway Networks*, where the output of a layer's neuron contains a combination of the input and the output. Highway networks use carry gates inspired by long short-term memory (LSTM) recurrent neural networks (RNNs) to regulate how much of the input is carried to the next layer. The authors demonstrate that their structure permits training networks of hundreds of layers (up to 900 layers) . These new parameters are learned along with the other parameters of the network. Zhang applied highway networks to LSTM recurrent neural networks. DropIn is a simpler approach than highway networks as it does not contain gate parameters that need to be learned.

Breuel discusses a dynamic network that he describes as a biologically plausible "reconfigurable" network. In this network different units are weighted dynamically to produce different configurations. This allows a single network to perform multiple tasks. DropIn represents a different type of dynamic network that grows during training rather than reconfigures for each task.

## Regularization during training

The well-known dropout method is an effective means to improve the training of deep neural networks. During training dropout randomly zeros a neuron's output activation with a probability $p$, called the *dropout ratio*, so that the network cannot rely on a particular configuration. This reduces overfitting to the training data and the resulting network is more robust and better generalizes to unseen data. While dropout "samples from an exponential number of different 'thinned' networks" , DropIn samples from an exponential number of different thinner and shallower sub-networks. Like dropout, DropIn randomly changes the configuration so that the network cannot rely on a particular configuration.

Baldi and Sadowski provide a theoretical basis for understanding dropout, demonstrating that dropout regulates the training and prevents overfitting by approximating an average of a large ensemble of networks. A similar theoretical understanding (and benefits) can also apply to DropIn.

# DropIn method

In this section we provide a mathematical basis for DropIn as well as some implementation details.

## Model description

There are two modes of running DropIn: first to gradually include skipped layers, which we refer to as *gradual DropIn*, and second as a regularizer, which we named *regularizing DropIn*. Figure provides a visual reference as to how the DropIn unit works.

Gradual DropIn initially passes on only the activations from the previous layer, effectively skipping the new layers. For each iteration number, $\tau$, the ratio $p$ is computed as $p = \tau / d$ for DropIn length $d$, which is the number of iterations over which $q = 1 - p$ reduces from 1 to 0. Then the number of activations copied from layer ${\ell-1}$ drops as $q \times n = (1 - p) \times n$, where $n$ is the total number of activations in the layer ${\ell-1}$. The remaining activations are accepted from the new layer ${\ell}$ and backpropagation trains the weights of these newly added units.

For regularizing DropIn, the DropIn probability ratio $p$ is set to a static value in $[0,1]$. In this case, DropIn works analogously with dropout but instead of setting values to zero, they are set to the activations of a previous layer (e.g., ${\ell-1}$). The choice of which activations come from which layer is done in an evolving random fashion each iteration.

We follow the notation in the dropout paper to show this more formally. Namely, we start with a neural network composed of some number of layers, $L$, where $\ell \in [1, 2, \dots, L]$ is the layer index. Also, $\mathbf{y}^{(\ell)}$ represents the vector of outputs from layer ${\ell}$ and is the input to the next layer ${\ell+1}$. Let $\mathbf{x}$ be the data input to the first layer. In addition, $\mathbf{W}^{(\ell)}$ and $\mathbf{b}^{(\ell)}$ are the weights and biases at layer $\ell$. To allow us to track the evolving nature of the network, we include the training iteration number, $\tau$, and the layer's unit index number, $\lambda^{(\ell)}$.

The first equation for gradual DropIn is a vector of zeros then ones, which is designated as: $$\begin{aligned}
\mathbf{r}^{(\ell)}(\tau,\lambda^{(\ell)}) =\begin{cases} 0 &\lambda^{(\ell)}< q \times n \\
1 & \mbox{otherwise}.
\end{cases}
\end{aligned}$$ For regularizing DropIn, the equation for $\mathbf{r}^{(\ell)}(\tau, \lambda^{(\ell)})$ with a probability ratio $p$ is: $$\mathbf{r}^{(l)}(\tau,\cdot) \sim Bernoulli(p),$$ i.e., a 0-1 vector where each value is distributed as a Bernoulli random variable with probability $p$.

Once $\mathbf{r}$ is set, the remaining equations (dropping $\tau$ and $\lambda^{(\ell)}$ for simplicity) are the same for both modes – namely for layer ${\ell+1}$: $$\mathbf{\tilde{y}}^{(\ell)} = \mathbf{r}^{(\ell)} \times \mathbf{y}^{(\ell)}  \\$$ $$z^{(\ell+1)}_i  = \mathbf{w}^{(\ell+1)}_i \mathbf{\tilde{y}}^{(\ell)} + b_i^{(\ell+1)} \\$$ $$\mathbf{y}^{(\ell+1)} = f ( z_i^{(\ell)} ) + ( 1 - \mathbf{r}^{(\ell)} )  \mathbf{y}^{(\hat{\ell})},$$ where $\hat{\ell}$ is any layer less than layer ${\ell+1}$. These equations are similar to those for dropout, except instead of some of the outputs being zero, they are set to the values from a previous layer, $\mathbf{y}^{(\hat{\ell})}$.

## Implementation

We implemented our method in Caffe by creating a new layer called DropIn. The parameters for the DropIn layer include a $dropin\_ratio$, $q = 1 - p$ (Figure ), and a $dropin\_length$, $d$, as described in Section .

DropIn requires that the size of both the new layer and the previous layer be the same. Hence, we also implemented a Resize layer to allow reshaping a layer's output to a user-specified size. The Resize layer modifies its input, which is $\mathbf{y}^{(\hat{\ell})},$ into a user-specified height, width, and number of channels/filters. The Resize layer allows DropIn to work with any two layers, even when the sizes of $\mathbf{y}^{(\ell)}$ and $\mathbf{y}^{(\hat{\ell})}$ are different.

# Experiments

The purpose of this section is to demonstrate the effectiveness of DropIn on several standard datasets but with architectures that are not trainable with standard methods. No attempt was made to optimize the architecture or hyper-parameters for higher accuracy because our main objective was to show that a deep architecture that will not converge without DropIn, will converge with it. However, the results in Sections and also demonstrate an increase in accuracy by using a deeper network for Imagenet.

All of the following experiments were run with Caffe (downloaded August 31, 2015) using CUDA 7.0 and Nvidia's CuDNN. These experiments were run on a 64 node cluster with 8 Nvidia Titan Black GPUs, 128 GB memory, and dual Intel Xenon E5-2620 v2 CPUs per node.

The following subsections depict, in table form, the structure of several networks. We use the naming convention {layer type}{layer number}-{number of outputs}(filter size). For example, conv1_2-32$(5\times 5)$ represents a convolutional layer numbered 1_2 with 32 outputs and filters sized $5\times 5$. DropIn layers are denoted as dropin $(\ell + (\ell + 1))$, as depicted in Figure .

In Section we show that very deep networks are trainable when using gradual DropIn on expanded LeNet with MNIST data. In Section we show the effect of DropIn length on training accuracy for expanded CIFAR-10 network and that a small performance gain is possible with added layers. In Section we show that an expanded AlexNet architecture increases accuracy and is trainable only with gradual DropIn . In Section the VGG16 network is trained using gradual DropIn without the need to transfer weights from a shallower network.

|          LeNet          |   LeNet(2N) + DropIn    |
|:-----------------------:|:-----------------------:|
|  data $(28\times 28 )$  |                         |
| conv1_1-20$(5\times 5)$ | conv1_1-20$(5\times 5)$ |
|                         | conv1_2-20$(3\times 3)$ |
|                         |   dropin (1_1 + 1_2)    |
|                         | conv1_3-20$(3\times 3)$ |
|                         |   dropin (1_2 + 1_3)    |
|                         |        $\vdots$         |
|                         | conv1_N-20$(3\times 3)$ |
|                         | dropin (1\_(N-1) + 1_N) |
|   maxpool$(2\times2)$   |                         |
| conv2_1-50$(5\times 5)$ | conv2_1-50$(5\times 5)$ |
|                         | conv2_2-50$(3\times 3)$ |
|                         |   dropin (2_1 + 2_2)    |
|                         | conv2_3-50$(3\times 3)$ |
|                         |   dropin (2_2 + 2_3)    |
|                         |        $\vdots$         |
|                         | conv2_N-50$(3\times 3)$ |
|                         | dropin (2\_(N-1) + 2_N) |
|   maxpool$(2\times2)$   |                         |
|         fc3-500         |                         |
|         fc4-10          |                         |
|        soft-max         |                         |

Network architecture for LeNet and LeNet(2N)+ DropIn.

## MNIST

This dataset consists of 70,000 grey-scale images with a resolution of 28x28[^1]. Of these, 60,000 are for training and 10,000 are for testing. There are ten classes, each a different handwritten digit from zero to nine, with 7,000 images per class.

The standard network architecture for the classification of MNIST, provided in the Caffe package, is the 4-layer LeNet consisting of 2 convolutional/max-pooling layers followed by 2 fully-connected layers (see the first column of Table for details). Inspired by the work in , we increased the number of convolutional layers from two to 2N, which we denote as LeNet(2N). These added layers (as seen in the second column of Table , minus the DropIn layers shown in red) learned a $3\times 3$ convolution filter but did not change the size of the outputs. We then added DropIn layers between each of the convolutional layers (as seen in the second column of Table ) and called this network LeNet(2N) + DropIn.

We first looked at $N=5$ and created LeNet(10) and LeNet(10) + DropIn architectures. LeNet(10) did not converge in the standard training time of 10,000 iterations given multiple realizations of the training process. However, utilizing DropIn units we were able to have LeNet(2N) + DropIn converge 10,000 iterations with the same hyper-parameters. In Figure we show results for several different DropIn lengths for this network. These different lengths indicate the robustness of the DropIn length for simpler networks and that, in general, shorter DropIn lengths provide marginally better results. We note for this case that the added layers do not increase the overall accuracy of the network, as the MNIST data is quite simple compared with other classification tasks; the added layers do not provide any extra differentiation power.

We now look at how the number of layers affects the training with DropIn. In Figure there are two different plots, one with DropIn length of 2,500 iterations and the other with DropIn length of 7,500 iterations. For each plot we present three different networks with 10, 30, and 50, convolutional layers (equating to N=5, 15, 25). For both DropIn lengths and all three network depths, the gradual DropIn method allowed the networks to converge. The deeper networks require a greater number of iterations to reach the same level of accuracy as the shallower networks, which is to be expected as they have a greater number of weights to train. We also see that networks converge more quickly with the shorter DropIn length, indicating that shorter DropIn lengths are desirable.

|          CIFAR-10          | CIFAR-10(11 layers) + DropIn  |
|:--------------------------:|:-----------------------------:|
| data $(32\times32\times3)$ |                               |
|   conv1-32$(5\times 5)$    | conv1_1-32$(5\times 5)$ + LRN |
|    maxpool$(2\times2)$     | conv1_2-32$(5\times 5)$ + LRN |
|            LRN             |      dropin (1_1 + 1_2)       |
|   conv2-32$(5\times 5)$    | conv2_1-32$(5\times 5)$ + LRN |
|    maxpool$(2\times2)$     | conv2_2-32$(5\times 5)$ + LRN |
|            LRN             |      dropin (2_1 + 2_2)       |
|                            | conv3_1-32$(5\times 5)$ + LRN |
|                            | conv3_2-32$(5\times 5)$ + LRN |
|                            |      dropin (3_1 + 3_2)       |
|                            | conv4_1-32$(5\times 5)$ + LRN |
|                            | conv4_2-32$(5\times 5)$ + LRN |
|                            |      dropin (4_1 + 4_2)       |
|                            | conv5_1-32$(5\times 5)$ + LRN |
|                            | conv5_2-32$(5\times 5)$ + LRN |
|                            |      dropin (5_1 + 5_2)       |
|   conv3-64$(3\times 3)$    |    conv6_1-64$(3\times 3)$    |
|    maxpool$(2\times2)$     |                               |
|           fc-10            |                               |
|          soft-max          |                               |
|          accuracy          |                               |

CIFAR-10 11-layer architecture, including DropIn units.

| Architecture | dropin_length | Accuracy (%) |
|:------------:|:-------------:|:------------:|
| 3-layer net  |               |     81.4     |
| 11-layer net |     8,000     |     81.7     |
| 11-layer net |    16,000     |   **82.3**   |
| 11-layer net |    24,000     |   **82.3**   |

Final accuracy (average of last three values) results for the CIFAR-10 dataset on test data at the end of the training. Comparison of DropIn and dropin_lengths.

## CIFAR-10

This dataset consists of 60,000 color images with a resolution of 32x32. Of these, 50,000 are for training and 10,000 are for testing. There are ten classes with 6,000 images per class.

The Caffe website provides the architecture and hyper-parameter settings as part of the CIFAR-10 tutorial[^2]. The three convolutional layer architecture trains quickly and attains good accuracies. The convolutional layers were replicated to obtain an 11-layer model, which corresponds to the depth of one of the CIFAR-10 models in the experiments for highway networks . The detailed architectures are compared in Table . As shown in the table, the sizes of each of the layers entering the DropIn layer were kept the same for simplicity. For every convolutional layer, the weight initialization was Gaussian with standard deviation of 0.01 and the bias initialization was constant, set to 0. Each convolutional layer was followed by a rectified linear unit and local normalization. The length of the training, the learning rates, and schedule were modified to run over 32,000 iterations. This modification trained satisfactorily and provided a reasonable comparison.

Numerous attempts at training this 11-layer network without the DropIn layers failed to converge. Similar attempts to train this network with the DropIn layers did successfully converge, which is a primary result of this study.

Experiments were performed varying the DropIn length. Figure shows the accuracy curves for $dropin\_length = 8,000, 16,000, 24,000$, and Table compares the final accuracies. The final accuracies show a marginal improvement for longer versus shorter lengths but for CIFAR-10 the results are relatively independent of the length value. Furthermore, the final accuracies from the 11-layer architecture are less than 1% better than the original 3-layer architecture, which implies that for the CIFAR-10 dataset, the deeper networker provides only marginal improvement.

|             AlexNet             | AlexNet (13 layers) + DropIn  |
|:-------------------------------:|:-----------------------------:|
|  data $(227\times227\times3)$   |                               |
|    conv1_1-96$(11\times 11)$    |   conv1_1-96$(11\times 11)$   |
|                                 |   conv1_2-96$(11\times 11)$   |
|                                 |      dropin (1_1 + 1_2)       |
| maxpool$(2\times2)$ + LocalNorm |                               |
|    conv2_1-256$(5\times 5)$     |   conv2_1-256$(5\times 5)$    |
|                                 |   conv2_2-256$(5\times 5)$    |
|                                 |      dropin (2_1 + 2_2)       |
| maxpool$(2\times2)$ + LocalNorm |                               |
|    conv3_1-384$(3\times 3)$     |   conv3_1-384$(3\times 3)$    |
|                                 |   conv3_2-384$(3\times 3)$    |
|                                 |      dropin (3_1 + 3_2)       |
|    conv4_1-384$(3\times 3)$     |   conv4_1-384$(3\times 3)$    |
|                                 |   conv4_2-384$(3\times 3)$    |
|                                 |      dropin (4_1 + 4_2)       |
|    conv5_1-256$(3\times 3)$     |   conv5_1-256$(3\times 3)$    |
|                                 |   conv5_2-256$(3\times 3)$    |
|                                 |      dropin (5_1 + 5_2)       |
|       maxpool$(2\times2)$       |                               |
|            fc6-4096             |                               |
|            fc7-4096             |                               |
|            fc8-1000             |                               |
|            soft-max             |                               |

Network architecture for AlexNet and modified version of AlexNet, AlexNet (13 layers) + DropIn .

|    Architecture    | dropin_length | Accuracy (%) |
|:------------------:|:-------------:|:------------:|
|      AlexNet       |               |     58.0     |
| 13 layers + DropIn |    25,000     |   **62.2**   |
| 13 layers + DropIn |    75,000     |     62.1     |
| 13 layers + DropIn |    150,000    |     60.8     |
| 13 layers + DropIn |    300,000    |     59.3     |

Comparison of DropIn and dropin_lengths, $d$. The table shows final accuracy (average of last three values) results for the ImageNet dataset on validation data at the end of the training.

## ImageNet / AlexNet

ImageNet[^3] is a large image database based on the nouns in WordNet. This image database, used for the ImageNet Large Scale Visual Recognition Challenge, is commonly used as a basis of comparison in the deep learning literature. The database contains 1.2 million training and 50,000 testing images covering 1,000 categories.

The Caffe website provides the architecture and hyper-parameter files for a slightly modified AlexNet. We downloaded the architecture and hyper-parameter files from the website and we expanded the architecture from 8 layers to 13 layers by duplicating each of the convolutional layers, which is shown (minus the DropIn layers shown in red) in columns 1 and 2, respectively, of Table . The AlexNet (13 layers) + DropIn includes a DropIn layer between every duplicated layer used to create AlexNet (13 layers). Multiple attempts at training the AlexNet (13 layers) architecture in the conventional manner did not converge. In the tests with the expanded architecture, the hyper-parameters were kept the same as provided by the Caffe website (even though our experiments with DropIn indicate that tuning them could improve the results).

Experiments were run varying the DropIn hyper-parameter *dropin_length*. Table shows final accuracy results after training for 450,000 iterations with a range of lengths. Figure compares the accuracy during training of these experiments. In contrast to the results with CIFAR-10, the DropIn length makes a difference with ImageNet. We believe that this is because the deeper architecture increases the classification accuracy for larger datasets, hence the improvement with smaller DropIn lengths is more prominent.

From Figure and Table , we can conclude that shorter lengths are better than the longer ones. If the length is less than the first scheduled drop in the learning rate at iteration 100,000, then the network is better trained. However, the difference between $dropin\_length = 75,000$ and 25,000 is negligible implying that lengths less than the first scheduled learning rate drop are equivalent.

|             VGG8             |      VGG16 + DropIn      |
|:----------------------------:|:------------------------:|
| data $(224\times224\times3)$ |                          |
|   conv1_1-64$(3\times 3)$    | conv1_1-64$(3\times 3)$  |
|                              | conv1_2-64$(3\times 3)$  |
|                              |    dropin (1_1 + 1_2)    |
|     maxpool$(2\times2)$      |                          |
|   conv2_1-128$(3\times 3)$   | conv2_1-128$(3\times 3)$ |
|                              | conv2_2-128$(3\times 3)$ |
|                              |    dropin (2_1 + 2_2)    |
|     maxpool$(2\times2)$      |                          |
|   conv3_1-256$(3\times 3)$   | conv3_1-256$(3\times 3)$ |
|                              | conv3_2-256$(3\times 3)$ |
|                              |    dropin (3_1 + 3_2)    |
|                              | conv3_3-256$(3\times 3)$ |
|                              |    dropin (3_2 + 3_3)    |
|     maxpool$(2\times2)$      |                          |
|   conv4_1-512$(3\times 3)$   | conv4_1-512$(3\times 3)$ |
|                              | conv4_2-512$(3\times 3)$ |
|                              |    dropin (4_1 + 4_2)    |
|                              | conv4_3-512$(3\times 3)$ |
|                              |    dropin (4_2 + 4_3)    |
|     maxpool$(2\times2)$      |                          |
|   conv5_1-512$(3\times 3)$   | conv5_1-512$(3\times 3)$ |
|                              | conv5_2-512$(3\times 3)$ |
|                              |    dropin (5_1 + 5_2)    |
|                              | conv5_3-512$(3\times 3)$ |
|                              |    dropin (5_2 + 5_3)    |
|     maxpool$(2\times2)$      |                          |
|           fc6-4096           |                          |
|           fc7-4096           |                          |
|           fc8-1000           |                          |
|           soft-max           |                          |

Network architectures for VGG8 and VGG16 + DropIn. See the text for additional settings.

## ImageNet / VGG

VGG$n$, a set of networks created by the Visual Geometry Group , won second place in the image classification category of the 2014 ImageNet contest. These networks, trained on the same database as the Alexnet architecture discussed in Section , contained $n=11,13,16,\mbox{ or }19$ layers. In Table we see the VGG16 (minus the DropIn layers shown in red) architecture alongside what we will refer to as VGG8 (not contained in the original paper). All convolutional layers have a stride and padding of 1 and maxpooling layers have a stride of 2. In their paper, the authors describe the difficulty of training these deep networks and utilized a weight transfer method to enable the network to converge during training .

While it is possible to train a deep neural network by first training a shallow network and using those weights to initialize the deeper network, we believe that in addition to being easier, training the full network with all the layers in place leads to a better trained network. This is supported by research on feature visualization, such as in Zeiler and Fergus , where they demonstrate that higher layers have more abstract representations. Training in place means that the learned representations will conform well to the representation at a given layer, while training a shallow network and initializing the weights of a deeper network might not.

Instead of training smaller networks, we propose to use our gradual DropIn method. For our studies, we utilized the VGG16 prototxt file referenced on the Caffe website[^4] and set up the solver file with the appropriate parameters from the authors' paper. Using traditional training methods, we were only able to train the VGG8 architecture; the VGG16 failed to begin converging for multiple realizations. Using VGG8 as a template, we augment VGG16 with DropIn layers to create VGG16 + DropIn (see Table ).

Based on the evidence presented in Section , we choose to test VGG16 with a DropIn length of 60,000. We found that other lengths (100,000, 150,000, and 200,000) began to converge as well but with limited time and resources, we chose to report only this length for this paper. The results of training VGG16 + DropIn are shown in Figure alongside VGG8. We see that with gradual DropIn the difficult to train VGG16 network does converge. Here we see the real power of the gradual DropIn method; without training an additional shallower network we are able to directly train VGG16, thus saving effort for the practitioner.

| Case |   fc6   |   fc7   |
|:----:|:-------:|:-------:|
|  1   | dropout | dropout |
|  2   | dropout |         |
|  3   | dropout | DropIn  |

The three regularization experiments shows layers with dropout or DropIn . The fully connected layers 6 and 7, are called *fc6* and *fc7*, respectively.

## Using DropIn for regularization

The original AlexNet architecture uses dropout for regularization during training in both fully connected layers and it provides a substantial increase in the network's accuracy. AlexNet (with 8 layers) provides a means to test DropIn regularization. For this experiment, three cases were run as shown in Table . Case 1 is the original AlexNet.

The results from this experiment are shown in Figure , where both DropIn and dropout probability ratios were $0.5$ for all of these tests and all the other hyper-parameters were the same. This figure shows that removing dropout from fc7 causes visible degrading of the accuracy between iterations 150,000 and 200,000 (green curve). This kind of degradation does not happen with DropIn. Instead, the accuracy curve is similar to the curve with dropout (red versus blue curve) but with a small degradation in overall performance. We believe this degradation is because a DropIn network is more difficult to train than a dropout network. However, the final accuracy for the network with DropIn in fc7 is higher than from an architecture without dropout (red versus green curve). This experiment demonstrates that DropIn provides some regularization since the degradation found in the case without dropout is absent.

# How to determine a good architecture

One of the challenges for deep learning practitioners is to determine good choices for the hyper-parameter values and the architecture for a given application and dataset. DropIn and dropout provide an easier way to test choices for the architecture than running a set of experiments with many different architectures.

DropIn and dropout can allow one to test a range of architecture depths and widths, respectively. Since adding layers does not necessarily increase accuracy, one can run with the gradual DropIn mode to see if there is little effect, such as in Figures and , or visible effect, such as in Figure . Substantial improvement implies that there will be benefit from the additional depth.

Similarly, making a run where the dropout ratio varies from perhaps 0.9 to 0.1 (using a slightly modified dropout) provides guidance on the minimum number of neurons per layer. When decreasing the probability that neurons are retained (as shown in Figure 9 of Srivastava ), the error typically has a range of the probability ratios where the error plateaus but at some threshold probability the error increases. By multiplying the number of neurons in a layer by this threshold probability, one can approximately determine the minimum number of neurons one must retain where there is marginal harm to the accuracy.

# Conclusion

The major result of this paper is that deeper architectures that cannot converge using standard training methods, become trainable by slowly adding in the new layers during the training. In addition, there are indications that DropIn layers help regularize the training of a network. We found in general that if the shallow network is trainable, then the deeper network, where additional layers are added by a DropIn layer, is also trainable. With a large dataset like ImageNet, adding additional layers increases accuracy.

We have not yet explored training with tailored DropIn lengths for different DropIn layers in a network. In addition, comparing DropIn to initializing the weights from training a separate shallow network has not yet been tested; these are planned for future work and will be reported elsewhere. Also we plan to test DropIn within other architectures such as recurrent neural networks. Future work also includes training networks with hundreds of layers using asynchronous DropIn, where layers are added starting at different iterations. In addition, we wish to test training where the entire very deep network is initially very thin (few parameters to train) and units are added to all the layers during the training. Furthermore, we plan to study if a methodology can be developed to learn from the data how to automatically optimize the architecture during training and thus learn to adapt to an application based on its data.

[^1]: <http://yann.lecun.com/exdb/mnist/>

[^2]: <http://caffe.berkeleyvision.org/gathered/examples/cifar10.html>

[^3]: <a href="www.image-net.org/" class="uri">www.image-net.org/</a>

[^4]: <https://gist.github.com/ksimonyan/211839e770f7b538e2d8#file-vgg_ilsvrc_16_layers_deploy-prototxt>


abstract: Magnetic fields in galaxies are believed to be the result of dynamo amplification of initially weak seed fields, reaching equipartition strength inside the interstellar medium. The small-scale dynamo appears to be a viable mechanism to explain observations of strong magnetic fields in present-day and high-redshift galaxies, considering the extreme weakness of seed fields predicted by battery mechanisms or primordial fields.
  .
  Performing high-resolution adaptive mesh magneto-hydrodynamic simulations of a small mass, isolated cooling halo with an initial magnetic seed field strength well below equipartition, we follow the small-scale dynamo amplification from supernova-induced turbulence up to saturation of the field. We find that saturation occurs when the average magnetic pressure reaches only 3 % to 5 % of the turbulent pressure. The magnetic energy growth transitions from exponential to linear, and finally comes to halt. The saturation level increases slightly with grid resolution. These results are in good agreement with theoretical predictions for magnetic Prandtl numbers of order $\mathrm{Pr_M} \sim 1$ and turbulent Mach numbers of order $\mathrm{M} \sim 10$. When we suppress supernova feedback after our simulation has reached saturation, we find that turbulence decays and that the gas falls back onto a thin disk with the magnetic field in local equipartition.
  .
  We propose a scenario in which galactic magnetic fields are amplified from weak seed fields in the early stages of the Universe to sub-equipartition fields, owing to the turbulent environment of feedback-dominated galaxies at high redshift, and are evolved further in a later stage up to equipartition, as galaxies transformed into more quiescent, large spiral disks.
author: Michael Rieder[^1] and Romain Teyssier
Institute for Computational Science
Centre for Theoretical Astrophysics and Cosmology
Universität Zürich, 8057 Zürich, Switzerland
bibliography: michael.bib; romain.bib
title: A small-scale dynamo in feedback-dominated galaxies - II. The saturation phase and the final magnetic configuration

# Introduction

Measurements of Faraday rotation in the Milky Way , in nearby galaxies as well as in high-redshift galaxies reveal strong magnetic fields, usually close to equipartition with the turbulent energy density. have detected field strengths up to $18 \, \mathrm{mG}$ in starburst galaxies but ordered galactic magnetic fields in the ISM of normal spiral galaxies are typically of the order of several $\, \mathrm{\mu G}$. Their field lines mostly exhibit a spiral structure if the galaxy is itself a grand design spiral galaxy but, interestingly, this can also be the case for ring galaxies like NGC 4736 , flocculent galaxies like NGC 4414 without clear spiral arms or in the central regions of galaxies. Strong ordered fields are found at the edges of optical arms with dense cold molecular gas in M 51 but can also form their own magnetic arms not coinciding with the gaseous or the optical spiral arms like in NGC 6946 . In a set of aggregated data on 20 spiral galaxies from the literature, report pitch angles ranging between $-8\, \mathrm{^\circ}$ and $-48\, \mathrm{^\circ}$ with a mean value of $-25\, \mathrm{^\circ}$ and find a correlation between the spiral arm pitch angles and the magnetic pitch angles.

The origin of magnetic fields in the Universe might be primordial or due to microphysical processes at later epochs, such as the Biermann battery in shock fronts or ionization fronts , spontaneous fluctuations or fluctuations due to the Weibel instability in the plasma of protogalaxies, or even magnetic fields released into the ISM by stars through stellar winds or supernova outbursts or even by AGN jets and subsequently diluted.

Microphysical mechanisms (such as the Biermann battery) are capable of creating magnetic fields of the order of $10^{-20} \, \mathrm{G}$, while the constraints on the primordial field are less definite because the difference between lower and upper limits remains vast. Upper limits can be derived from Big Bang Nucleosynthesis (BBN) abundances and from the large-scale density structure or the Cosmic Microwave Background (CMB). currently set the upper limit of the primordial magnetic field (PMF) field strength to $10^{-9} \, \mathrm{G}$ in the comoving frame based on their CMB anisotropy measurements. $\gamma$-ray observations of blazar spectra give lower limits for the field strength ranging from $10^{-18} \, \mathrm{G}$ up to $10^{-15} \, \mathrm{G}$ , based on the remnants of the PMF that are believed to thread the intergalactic medium. This leaves us to explain many orders of magnitude magnetic field amplification in a timeframe of just a few Gyr.

Large-scale dynamos (LSD) are a viable mechanism to amplify magnetic fields coherently on (as the name suggests) large spatial scales. Theoretical models for galactic LSD exhibit exponential growth rates of the order of $\Gamma \simeq 0.01 - 0.1 \Omega$ , where $\Omega$ is the galactic angular rotation rate. This translates into an e-folding time scale of roughly 1 Gyr in a typical present-day spiral, making it virtually impossible to amplify the field as fast as required by the observations. proposed also a cosmic-ray-driven LSD, that was simulated for the first time by , who found a larger growth rate of $\Gamma \simeq \Omega$.

Small-scale dynamos (SSD) on the other hand, can have very fast magnetic field amplification, with timescales of the order of the eddy turnover times of the smallest turbulent eddies . The theoretical foundation of this dynamo theory is commonly attributed to . considered the Kolmogorov power spectrum of small-scale velocity fluctuations for the galactic dynamo and found that the magnetic energy spectrum scales with the wavenumber as $k^{3/2}$ on scales larger than the resistive scale. presented a theoretical analysis of the SSD, in the limits of both small and large magnetic Prandtl numbers, finding that its growth rate scales with respectively the magnetic or kinetic Reynolds number. They also evaluated the ratio of magnetic to turbulent energy after saturation, finding values between $0.1 \%$ and $50\%$, depending on the model of turbulence, on the value of Pr$_M$ and on the value of the Mach number. Their results are confirmed by previous numerical investigations on the turbulence-driven dynamo such as or , who also reported that the saturation level is slightly increasing with resolution. The main issue with the SSD mechanism is, however, that it creates strong fluctuating fields, for which the large scale component is subdominant, and generally does not reach equipartition. These two properties are in contradiction with observational data of nearby galaxies . On the other hand, it is plausible that the magnetic fields we observe in galaxies are not the result of just one single process, but probably a combination of various mechanisms, such as the two dynamos theory or various reconnection processes during the hierarchical assembly of galaxies.

The importance of feedback processes has been recognised increasingly over the past decade in galaxy formation theory , along with the influence of associated galactic winds and the dominant cold stream accretion mechanism . Observations suggest that galactic winds are ubiquitous in star bursting local galaxies , as well as many "normal" high redshift galaxies . Abundance matching between dark matter haloes and observed central galaxies suggests considerably stronger feedback processes than previously considered for early galaxies to regulate star formation through cosmic evolution, especially at high redshift, to maintain such a low star formation efficiency . Interestingly, found a correlation between strong magnetic field signatures in the Faraday Depth spectrum and strong Mg II absorption, which is associated with superwinds from starbursting galaxies , suggesting a link between strong outflows and a high magnetisation of the intergalactic medium.

In this rather violent, feedback-dominated scenario, dwarf galaxies play a very important role. They are the dominant galaxy population at high redshift, probably responsible for the cosmic re-ionisation . They are also the progenitors of the Milky Way satellites, which are useful laboratories to test our current galaxy formation paradigm. For the latter, violent feedback mechanisms have also been invoked to explain the absence of cusp in the dark matter density profile, and the presence of a dark matter core in low surface brightness galaxies . Cosmological simulations of dwarf galaxies have been performed with strong feedback recipes, confirming in this case the formation of a dark matter core .

In galaxies, the velocity field on both small and large scales resulting from repeated giant feedback events can have a direct influence on the growth of the magnetic energy. Indeed, supernovae explosions in the Milky Way have been considered for quite a long time as a source of helical gas motions, promoting a large-scale $\alpha$-dynamo in the Galaxy . The Milky Way is a rather quiescent galaxy with moderate supernova activity but might have undergone different phases in its lifetime. We therefore want to study in this work the small-scale dynamo-induced growth of magnetic fields in dwarf galaxies under the influence of strong feedback, how it reaches saturation and how the magnetic field could evolve when feedback becomes weaker. In the first step, we are considering a feedback-dominated galaxy, with a high star formation rate and violent turbulent motions, together with large-scale galactic fountains or winds. As the magnetic field grows until the small-scale dynamo is saturated, this feedback is turned off in the second step to see how it evolves in a less turbulent, quiescent phase.

In recent years, several simulations of the magnetic fields evolution have been performed in the context of galaxy formation , These early studies were based on the "cooling halo" numerical set-up, and have achieved only moderate magnetic field amplification. An important shortcoming in these simulations was the absence of feedback , or the relative weakness of the feedback recipe used at that time . performed a simulation using a new developed MHD version of the GADGET code GADGET, using a divergence cleaning technique, and they observed a fast exponential growth of the magnetic field, which they attributed to a small-scale dynamo. Surprisingly, they did not include any explicit source of turbulence resulting in a relatively smooth flow, but reported nevertheless a very large growth rate.

More recently, also observed strong magnetic field amplification using the MHD version of the AREPO code , although also in this case, stellar feedback effects were not introduced explicitly, and the smoothness of their effectively 2D rotating flow would make dynamo amplification notoriously difficult to obtain. The same code was used very recently for cosmological zoom-in simulations of Milky Way-like disk galaxies in , this time including a more realistic treatment of stellar feedback, resulting in strong turbulence driving. As a consequence, a fast magnetic energy amplification was observed at high redshift, attributed to the SSD, followed by a slower amplification at lower redshift, attributed to the LSD.

In our previous paper , simulating a dwarf galaxy at high resolution, we found strong evidence for a small-scale dynamo operating in galaxies with feedback-driven turbulence, with e-folding timescales of up to 100 Myr. A similar approach was used in for a Milky Way galaxy where the magnetic field was seeded by supernova ejections. Another interesting study was conducted recently by who impose a spiral potential in their simulations of an isolated disk and found magnetic field reversals.

In the present paper, we follow up on our previous work , using again the isolated cooling halo setup, with strong stellar feedback to investigate the mechanism of magnetic field amplification through the SSD. We use the Adaptive Mesh Refinement code RAMSES , adopting the "Constrained Transport", strictly divergence-free-preserving, MHD solver presented in and in . We focus here our analysis of the saturation properties of the SSD in the galaxy formation context. The paper is organised as follows: In [Section ](#chap:num-methods), we present our numerical methods in terms of galaxy formation physics and magnetic fields evolution, as well as the intricacies of modelling realistic turbulent flows in numerical simulations. In [Section ](#chap:IC), we describe our initial conditions for the isolated, magnetised cooling dwarf halo. In [Section ](#chap:analysis), we present our main results on dynamo saturation and study the impact of resolution and stellar feedback. In [Section ](#chap:discussion), we discuss our results in the context of current galactic dynamo theories. Finally, in [Section ](#chap:conclusions), we discuss the implications of our work for our understanding of cosmic magnetism.

# Numerical Methods

We use the Adaptive Mesh Refinement (AMR) code RAMSES to simulate the formation and evolution of an isolated dwarf galaxy. It is modelled as a magnetised ideal plasma, coupled through gravity to a collisionless fluid made of dark matter and stars, with additional numerical schemes to account for physical sub-resolution processes such as gas cooling, star formation and supernova feedback. In this section, we outline the numerical methods used in this work to model self-gravitating supersonic turbulence driven by stellar feedback.

## Ideal MHD with Gravity

We solve the ideal MHD equations which are written here without gravity and cooling source terms for the sake of simplicity: $$\begin{aligned}
\partial_{t} \rho + \nabla \cdot (\rho \mathbfit{u}) &= 0 \\
\partial_{t} (\rho \mathbfit{u}) + \nabla \cdot ( \rho \mathbfit{u} \mathbfit{u}^{T} - \mathbfit{B} \mathbfit{B}^{T} + P _{\mathrm{tot}} ) &= 0 \\
\partial_{t} E + \nabla \cdot \left[ (E + P _{\mathrm{tot}}) \mathbfit{u} - (\mathbfit{u} \cdot \mathbfit{B}) \mathbfit{B} \right] &= 0 \\
\partial_{t} \mathbfit{B} - \nabla \times \left( \mathbfit{u} \times \mathbfit{B} \right) &= 0
\label{induction-eqn}
\end{aligned}$$ where $\rho$ is the gas density, $\rho \mathbfit{u}$ is the momentum, $\mathbfit{B}$ is the magnetic field, $E = \frac 1 2 \rho \mathbfit{u}^{2} + \rho \varepsilon + \frac 1 {2} \mathbfit{B}^{2}$ is the total energy, and $\varepsilon$ is the specific internal energy. The total pressure is given by $P_{tot} = P +
\frac{1}{2} \mathbfit{B}^{2}$ where we assume a perfect gas equation of state $P = (\gamma - 1) \rho \varepsilon$. This system of conservation laws is furthermore completed by the solenoidal constraint $$\label{solenoidal} \nabla \cdot \mathbfit{B} =0.$$

RAMSES uses a hybrid approach with gas variables stored on a tree-based adaptively refined mesh, while dark matter and stars are tracked by collissionless particles. The equations above are solved using the second-order unsplit Godunov scheme based on the MUSCL-Hancock method with the HLLD Riemann solver and MinMod slope limiter. The induction equation () is solved with the Constrained Transport (CT) method , which preserves the divergence of the magnetic field $\nabla \cdot \mathbfit{B} =0$ from the initial conditions, and MonCen sloper limiter. We use free-outflow boundary conditions with imposed zero-gradient at the simulation box boundaries for the gas variables and the perpendicular magnetic field.

To treat physical processes that are well below the resolution limit but nevertheless important in the theory of galaxy formation, we include several effects such as gas cooling, star formation and supernova feedback. Gas cooling is implemented using a standard H and He cooling function, with an additional metal cooling component, as in for temperatures above $10^{4}$ K, and fine-structure cooling of \[CI\] and \[OII\] below $10^4$ K, based on . Cooling stops at a minimum pressure floor in order to ensure the Truelove criterion and always resolve the Jeans length by at least 4 grid cells. We create star particles as a random Poisson process compliant with a Schmidt law as in . The effect of supernovae is modelled by releasing non-thermal energy into the ISM over a dissipation time scale of 20 Myr . We refer the interested reader to for a more detailed account.

## Turbulence

Modelling turbulence properly is central to our goal of simulating the small-scale dynamo. Two key quantities in this context are the Reynolds number, which is the ratio of inertial vs. viscous forces $$\mathrm{Re} = \frac{u L}{\nu}$$ where $\nu$ is the kinematic viscosity and $u$ and $L$ are the typical velocity and length scale of the problem, and its counterpart for the magnetic field, the magnetic Reynolds number as the ratio of induction vs. diffusion $$\mathrm{Re_M} = \frac{u L}{\eta}$$ where $\eta$ is the magnetic diffusitivity. Additionally, it is useful to define the magnetic Prandtl number $$\mathrm{Pr_M = \frac{Re_M}{Re}} = \frac{\nu}{\eta}$$ as the ratio of those two quantities and thus as the ratio of kinematic viscous and magnetic diffusivity. Our simulations do not account for microscopic diffusion processes, as we aim at solving the ideal MHD equations, but the stability of the numerical solution and its convergence towards the weak solution of the underlying model equations are both enforced by the numerical diffusion inherent to the Godunov scheme. Due to our limited spatial resolution of $\Delta x \simeq 10 \, \mathrm{pc}$, the numerical effective viscosity and magnetic diffusivity coefficients are both much greater than the typical physical values for the typical ISM.

RAMSES has been tested to successfully reproduce idealised fast dynamo flows such as the ABC flow or the Ponomarenko dynamo . The numerical magnetic Reynolds number was shown to be proportional to the inverse square of the number of grid points, which is to be expected with second-order schemes and rather smooth solutions. However, in the context of a highly complex flow such our turbulent galaxy, where kinematics is dominated by rotation and supernova explosions inside a deep gravitational potential, it becomes virtually impossible to exactly determine neither viscosity nor diffusion due to the numerical scheme. This is even further complicated by the adaptively refined mesh. Qualitatively, it is sufficient to know that viscous effects as well as magnetic reconnection and diffusion caused by the numerical scheme happen at a length scale which is close to the mesh size, which is mainly the cell size at the maximal refinement level. We also point out that numerical viscosity and numerical diffusion are approximately equal due to their identical numerical origin, so that the effective magnetic Prandtl number $\mathrm{Pr_M} \simeq 1$ .

# Initial Conditions

As in the first paper of this series , we simulate an isolated dark matter halo with gas cooling and a small initial rotation. In our previous work, we have conducted a study on varying initial conditions with different halo sizes and initial magnetic field topologies and we found no significant impact on the nature of the dynamo itself. If the feedback mechanism is strong enough to stir strong turbulence in the disk and launch a large scale galactic wind, the magnetic field lines quickly become mangled, so that their initial topology and symmetry are lost and the flow develop a strong characteristic random, quasi isotropic and mostly small-scale component. We are now especially interested in the final phase of the dynamo mechanism, when the magnetic energy growth reaches saturation.

|       parameter | value | unit                |
|----------------:|------:|:--------------------|
|       $R_{200}$ |    50 | kpc                 |
|       $V_{200}$ |    35 | km/s                |
|       $M_{200}$ |    14 | $10^{9}$M$_{\odot}$ |
|             $c$ |    10 |                     |
|       $\lambda$ |  0.04 |                     |
|   $f_{\rm gas}$ |    15 | %                   |
|         $T_{*}$ |   100 | K                   |
|  $\epsilon_{*}$ |     1 | %                   |
| $\eta_{\rm SN}$ |    10 | %                   |
|   $Z_{\rm ini}$ |  0.05 | Z$_{\odot}$         |
|     metal yield |    10 | %                   |
|   $B_{\rm max}$ |  0.35 | nG                  |
| sim. box length |   300 | kpc                 |

Parameters of initial halo set-up and additional physics mechanisms

The initial dark matter and baryonic matter densities follow the NFW profile with concentration parameter $c=10$ and spin parameter $\lambda = 0.04$ as in . Dark matter is sampled by $10^6$ particles and set to be in equilibrium with the gas by the density-potential pair approach of and . The temperature profile is also initialised to be in hydrostatic equilibrium. The numerical values of the parameters specific to this set-up are given in . In order to study the effect of numerical resolution on the dynamo saturation properties, we increase the resolution in a second simulation run. The resolution parameters for the two runs are given in .

|            |               |           |                 |                 |          |
|-----------:|:-------------:|:---------:|:---------------:|:---------------:|:--------:|
| resolution | $l_{\rm max}$ | $\Delta$x |  $m_{\rm res}$  |     $m_{*}$     | $n_{*}$  |
|            |               |  \[pc\]   | \[M$_{\odot}$\] | \[M$_{\odot}$\] | \[H/cc\] |
|        low |      14       |    18     |      1523       |      2108       |    14    |
|       high |      15       |     9     |       190       |       264       |   112    |

Size of smallest grid cells and mass resolutions

In , we have studied the dynamo only in the kinematic phase, choosing an arbitrarily small value for the initial magnetic energy. Now we want to see what happens when the magnetic field becomes strong enough to become dynamically important. We have therefore to increase the initial magnetic field strength, but keeping it small enough for the dynamo to operate in the kinematic phase for some time (so that the initial magnetic energy is several orders of magnitude smaller than equipartition), but not too small for the dynamo to saturate in a reasonable amount of time.

Initialising the magnetic field on an adaptively refined grid is a non-trivial task. In order to satisfy the zero-divergence constraint (), it would be tempting to simply set the initial field to a constant, as $$\mathbfit{B}_{\mathrm{0}} = \begin{pmatrix}0\\0\\B_{0}\end{pmatrix}$$ throughout the whole simulation box . However, this approach can lead to some numerical problems, as the Alvén wave speed $$v_A = \frac{B}{\sqrt{4\pi\rho}}$$ becomes very large at the simulation box boundary, where the density falls off by many orders of magnitude, and the magnetic energy accreted at late time will be much larger, comparatively to the gas internal or gravitational energy, than the magnetic energy accreted at early time. A constant magnetic field would also not be realistic, since a frozen-in magnetic field following the collapse of the hot gas in hydrostatic equilibrium into a dark matter halo should scale as $\left\vert\mathbfit{B} \right\vert \propto \rho^{2/3}$.

Combined with the aforementioned solenoidality constraint, this requires a more complex field topology. For this purpose, as in , we define the vector potential $$\mathbfit{A}_{\mathrm{0}} = B_{0}\left[\frac{\rho(r,z)}{\rho_{0}}\right]^{2/3}  r \mathbfit{e} _{\mathrm{\phi}}$$ where $\rho(r,z)$ is the initial gas density given by the NFW profile and $\mathbfit{e} _{\mathrm{\phi}}$ is the unit vector along the toroidal direction. The initial magnetic field is then set as the curl $$\mathbfit{B}_0 = \nabla \times \mathbfit{A}_0,$$ so that it has exactly zero divergence by design. The corresponding magnetic field has a vertical component which is symmetric with respect to the mid plane, while its radial component is antisymmetric so that its shape resembles that of a dipole.

# Results

The gas, though initialised in hydrostatic equilibrium, immediately starts to cool radiatively and loses thermal energy. The spherical structure collapses onto a rotationally supported disk in only a few Myr. This causes the gas disk to fragment into clumps denser than the star formation threshold and the first stars form. Their supernova explosions drive outflows that reach out several kpc above and below the galactic midplane and quickly, a self-regulated galactic fountain is established, where dissipative processes such as cooling and shocks balance off kinetic energy injection through supernovae explosions. The resulting kinematic properties are in very good agreement with those of observed nearby isolated dwarf galaxies .

This galactic fountains turns the whole galaxy into a giant "washing machine" that maintain a very high level of turbulence, injected on very large scales, namely the scale of the entire galaxy. Line-of-sight projections of density, velocity and temperature are plotted in . The small-scale structure with clumps and filaments is clearly visible. Hot bubbles resulting from SN explosions, with temperature between $10^5$ and $10^6$ K, rise until the gas can cool down and falls back to the midplane, so that instead of ordered rotation, we see a highly turbulent velocity field configuration.

Since the gas is not isothermal, the flow Mach number ${\cal M} = v/c_{\rm s}$, where $v$ is the local gas velocity and $c_s$ is the local sound speed, is a non-trivial parameter to determine. Nevertheless, we would like to make a good estimate of the typical Mach number in our flow. We plot in a mass-weighted histogram of the Mach number for every cell. We find that the flow is highly supersonic, with most of the gas mass at the Mach number in the range of ${\cal M} \approx 6 - 7$.

## Saturation of the small-scale dynamo

As demonstrated in our previous work , the strong turbulence injected on large scale by the galactic fountain triggers a small-scale dynamo that amplifies the magnetic field exponentially fast. The time evolution of the total magnetic energy inside the whole simulation box is given in , using both a linear and a logarithmic scale. The initial simulation phase is characterised by an exponential growth, typical of a fast dynamo in the kinematic regime, for which the field strength is too weak to have an effect on the flow. We measured a best-fit exponential growth rate of $\Gamma \simeq 3.3 \, \mathrm{Gyr}^{-1}$. This exponential amplification continues until the magnetic field becomes strong enough for the Lorentz force to back-react on the velocity field: the dynamo enters its non-linear phase. At that point, which is after $t_{\mathrm{nl}} \simeq 2.3 \, \mathrm{Gyr}$, the growth rate of magnetic energy becomes weaker and transitions into the onset of the saturation phase, where the growth is now linear in time, and not exponential anymore. Eventually, the field strength becomes so high that the dynamo saturates completely and the field amplification stops, in our simulation after a time of $t_{\mathrm{sat}} \simeq 3.8 \, \mathrm{Gyr}$.

In order to study the saturation level of the magnetic field, it is essential to measure the kinetic energy density of the turbulence. Defining and computing the turbulent energy in a realistic galactic environment is however not as easy as in periodic boxes with forced or decaying turbulence. In our case, gas motions are dominated by the ordered galaxy rotation and quasi-random supernova explosions, both confined inside the stratified gravitational potential of the combined disk and dark halo system.

As a proxy for turbulent energy, we use here the kinetic energy density of the velocity component perpendicular to the disk $$P_{\rm turb} \sim \frac 1 2 \rho u_z^2$$ since it is not affected by galactic rotation. The time evolution of the thermal, turbulent and magnetic energies, averaged in a cube of 3 kpc in the centre of the galaxy, is shown in . One can see that thermal and turbulent pressures are of the same order of magnitude, at approximately $10^{-12}$ erg/cc during the entire simulation. The magnetic pressure, on the other hand, defined as $P_{\rm mag} = \frac 1 {8\pi} B^2$, increases first exponentially, and then, at the onset of the non-linear dynamo phase, reaches only one per mil of the turbulent energy, corresponding also to a plasma $\beta = P_{\rm therm}/P_{\rm mag} \simeq 700$. At the end of the simulation, when the dynamo is fully saturated, the average magnetic energy reaches a strength of $10^{-14}$ erg/cc, which is 2.5% of the turbulent pressure, corresponding also to $\beta \simeq 40$.

shows volume-averaged line-of-sight projected maps of the different pressure components. One can clearly see that both thermal and turbulent pressure are relatively diffused and homogeneous, with only a few hot bubbles associated to supernova explosions appearing as distinctive features. The magnetic pressure map, on the other hand, is much more structured, with some striking filamentary features, resulting from a complex dynamo process. The magnetic pressure inside these filaments exceeds $10^{-12} \, \mathrm{barye}$, corresponding to a field strength $B > 1 \, \mathrm{\mu G}$.

We additionally plot the ratio of the magnetic to turbulent energy in (left panel) to evaluate to strength of the field at saturation. Although the magnetic energy (or pressure) averaged over some volume is always (sometimes significantly) below equipartition (even after the saturation, as shown in ), we can now see that the field has actually reached equipartition in localised regions, usually associated with these strongly magnetised filaments.

We plot in the spectra of the magnetic and kinetic energy inside the galaxy. The kinetic energy spectrum stays roughly constant over time and exhibits a characteristic power-law behaviour $E_{\mathrm{kin}} \propto k^\alpha$ with a best fit for $\alpha=-1.8$, which is between the theoretical values of $\alpha=-5/3$ for incompressible Kolmogorov turbulence and $\alpha=-2$ for highly compressible, shock-dominated Burgers turbulence. The magnetic energy spectrum develops also the characteristic shape predicted by Kazantsev's theory, with a power law on larger scales with index $1.5$, bottlenecked at small scales because of magnetic diffusion. During the kinematic phase, the magnetic energy power spectrum peaks at a length scale $\ell_{\rm peak} \simeq 500$ pc, and progressively shifts to larger scales until it reaches saturation, with a final peak length scale of $\ell_{\rm peak} \simeq 1 \, \mathrm{kpc}$. This rather large spatial scale is a fundamental prediction of our theoretical calculations, and is justified by the very large spatial scale (around 10 kpc or more) at which the galactic fountains injects turbulent energy into the magnetic dynamo.

Interestingly, our saturated dynamo being strongly coupled to a large scale galactic fountain, we are in a position to provide strong observational predictions on the strength of the magnetic field around high-redshift galaxies. Indeed, magnetic fields in the circumgalactic medium can be detected by the Faraday rotation of polarised emission from background sources, with impact parameters as large as the virial radius of the parent halo . We plot in the radial profiles of the mean gas density, the gas metallicity and the magnetic field strength averaged in spherical shells around the galaxy out to the virial radius at 30 kpc. We can see that the galactic fountain is polluting the IGM around the galaxy, with a metallicity of $Z \approx 0.2 Z_{\odot}$ and the magnetic field strength declining from $\mu$G to a tenth of a nG, and the gas density from $1 \, \mathrm{H/cc}$ to $\sim 10^{-4}$ H/cc.

## Effect of Resolution

The small-scale dynamo growth rate increases with the effective Reynolds number, which in our case is set by the numerical diffusion of our Godunov scheme. In order to study this numerical effect, we increase the resolution by one level to reduce both magnetic diffusion and kinematic viscosity, so that both effective Reynolds numbers (magnetic and kinematic) become larger. Due to limited computational resources, we could only run this simulation for 3 Gyr. However, since the dynamo operates more than twice as fast at this increased resolution, it reached saturation before the lower resolution run.

The magnetic energy evolution for this high-resolution run is plotted alongside the low resolution evolution in . The behaviour is qualitatively the same for both. However, the growth rate in the kinematic phase is larger with $\Gamma \simeq 9.4 \, \mathrm{Gyr}^{-1}$ in the high resolution case. Consequently, the onset of the non-linear dynamo phase happens earlier. The overall magnetic energy level at saturation is slightly higher than in the low-resolution run, so that the magnetic field saturates at $5 \%$ of turbulent energy density, corresponding also to a final plasma $\beta \simeq 20$.

We plot in the kinetic and magnetic energy spectra from the last snapshot of the high resolution run, compared to the same spectra at saturation for the low resolution run. The high-resolution kinetic spectrum has the same turbulent slope as the low-resolution one, with however a slightly higher amplitude, owing to the slightly more efficient star formation efficiency and associated supernovae feedback. The magnetic spectra for the high resolution run also agrees well with the low resolution one at large scale, with however slightly more power on small scales, due to the decrease of the effective dissipation length. This results in a slight shift of the peak scale from $\ell_{\rm peak}=1$ kpc to $\ell_{\rm peak}=0.7$ kpc.

## Transition to Quiescence

We have seen in the previous section that in the active, early phase of its life, dominated by stellar feedback, a galaxy with a fully developed turbulent fountain can power a small-scale dynamo that efficiently amplifies an initially weal seed magnetic fields to saturation. Naturally, one might wonder what would happen to this sub-equipartition field, if feedback becomes weaker. Indeed, present-day galaxies like the Milky Way have thin and quiescent disks, with a modest level of turbulence and a small kinetic energy injection scale around 100 pc, traditionally associated to the thickness of the gas disk or to local supernova super-bubbles . For that purpose, we re-run the simulation from our snapshot at 4 Gyr but with supernova feedback turned off and let it evolve for a couple of Gyr.

In , we show images of our galaxy at time 5.6 Gyr, 1.6 Gyr after feedback has been switched off. Without feedback to drive the turbulence anymore, the galaxy has entered a quiescent phase where the gas has cooled down and collapsed into a thin, although clumpy, rotationally supported disk, with a clear spiral structure. The velocity field is dominated by a strong rotational component, anti-clockwise in this image, which means that the spirals are trailing.

The evolution of the thermal, turbulent and magnetic pressure components is shown with dashed lines in . The average magnetic energy density decreases only slightly when the disk enters this new phase and remains at a level of $10^{-14}$ erg/cc. At the same time, we can see a clear drop in thermal energy density and, less pronounced, in the turbulent component. The latter, however, since we use the vertical gas velocity component as a proxy for turbulence, is still contaminated by a vertically collapsing flow.

To characterise the magnetic field strength in our thin disk galaxy even further, we compare the magnetic pressure to the gas thermal and turbulent energy densities side-by-side, by plotting line-of-sight projections of all three pressure components in . The contamination of our turbulent energy proxy by the vertical collapse is clearly visible from the in-falling gas above the disk. While they all have significantly decayed inside the galactic corona (except for the vertical component of the velocity field), they all remain strong inside the disk. Notably, the energy density of the magnetic field is now in equipartition with the other two energies inside the galactic arms, exceeding $10^{-12} \, \mathrm{barye}$, which amounts to field strengths greater or around $1 \, \mathrm{\mu G}$. We have also plotted the ratio of the magnetic to turbulent pressure in the quiescent case in the right panel of . The ratio reaches unity inside the arms, meaning that the magnetic field is in fact *locally* in equipartition with turbulence and thermal pressure in the dense galactic arms of our thin disk.

In order to characterise the magnetic field topology, we plot in the top panel of the toroidal, radial and vertical components of the field, as a function of the disk height, averaged within cylindrical shells of different sizes, parallel to the disk plane. On can see that after the collapse of the fountain into a thin disk, the vertical component of the field cancelled almost entirely, leaving only a dominant toroidal component, with a clear even-symmetry with respect to the disc plane, and a very weak average radial component. We argue here that this quadrupole symmetry is a natural consequence of a random magnetic field collapsing into the midplane, with a cancellation of odd-symmetric modes and a strengthening of even-symmetric ones during the formation of the thin disk. On the other hand, one also clearly sees sign reversals in the average toroidal component, so that its average over the entire disk remains very small.

In , we plot the three magnetic field components as a function of radius, averaged over very thin cylindrical shells of thickness $H = 40$ pc. Inside the innermost region up to 1 kpc, the magnetic field is dominated by the toroidal field component with field strengths up to $1 \, \mathrm{\mu G}$. One can also see in the radial profile significant sign reversals of the toroidal component. These sign reversals are a clear relic of the small scale dynamo in the feedback-dominated corona prior to the collapse, and are also tightly connected to the spiral pattern. In the lower panel of , we show the standard deviation of the three field components, averaged within the same small cylinders. This quantity is a measure of the field strength at small scale, and reaches 1 $\, \mathrm{\mu G}$ for the toroidal component, roughly in equipartition with the mean field. The standard deviation for the vertical component is almost zero, while the small scale radial component has a field strength around 0.3-0.4 $\, \mathrm{\mu G}$.

To quantify and illustrate the relative strength of the radial and toroidal components better, we show the magnetic field line directions ${B}_{\bot}$ perpendicular to the line of sight in a face-on view of the average magnetic field in , together with a histogram of the pitch angle $$p_B = \, \mathrm{arctan} \,{\frac{B_r}{B_t}}.$$ which is a common observable when measuring galactic magnetic fields. The magnetic field is strongest inside the trailing main arms which also coincide with the dense gas arms. Its field lines are generally aligned with the arm structure in the arms and otherwise mostly pointing along the toroidal direction (sometimes parallel, sometimes anti-parallel). We find that the pitch angles are symmetrically distributed around a mean pitch angle of $\langle p_{\mathrm{B}} \rangle = -12.4 \, \mathrm{^\circ}$ with a standard deviation of $\sigma(p_{\mathrm{B}}) = 35.1 \, \mathrm{^\circ}$. Note that these negative pitch angles are a clear indicator of magnetic field alignment with the spiral structure of a trailing spiral galaxy and change sign because of the field reversals.

# Discussion

We have set up magnetohydrodynamical simulations of an isolated cooling dwarf halo which hosts a galaxy with strong turbulence driven by supernova feedback. The gas component features high Mach number flows with up to $M \approx 10$ and a kinetic energy spectrum with a characteristic slope between incompressible Kolmogorov and highly compressible shock-driven Burgers turbulence. Just as in our previous work , the magnetic seed field is quickly amplified by a small-scale dynamo. The magnetic energy spectrum has the typical bottlenecked $3/2$-slope on large scales which peaks near the resistive scale and falls off on smaller scales as predicted by dynamo theory . The global magnetic field time evolution in our experiment can be divided into three distinct phases:

1.  exponential growth (kinematic phase)

2.  constant growth (non-linear phase)

3.  zero growth (saturation)

During the exponential growth phase, the e-folding times are $\sim 300 \, \mathrm{Myr}$ with lower resolution and $\sim 100 \, \mathrm{Myr}$ with higher resolution. Since the dynamo growth rate increases with the Reynolds numbers of the flow, it could potentially become very large if we were not constrained by our limited computational resources. Estimating the Reynolds numbers in a simulation with numerical viscosity and diffusivity can be very difficult when not using explicit terms in the system of equations and when complicated flows are involved. As a lower limit, we can take the numerical diffusion of a first-order method which is given by $$\mathrm{{Re}_{N}} = 2 \frac L {\Delta x}$$ with the typical length scale of the system $L$ and the spatial resolution $\Delta x$. This would give values for $\mathrm{{Re}_{N}} = 100$ in the lower resolution case and 200 for the higher resolution in our simulations. Generally, in this kind of simulations, both the kinematic as well as the magnetic Reynolds number will range between a few $100$ and $1000$. We can compare this to a typical ISM with an estimate for the ISM viscosity $$\, \mathrm{Re} \,{=} \sqrt{3\gamma} M \frac L \lambda$$ where $\lambda$ is the mean free path of a hydrogen atom and $M$ is the Mach number. At a density of $1 \, \mathrm{H/cc}$ and a Mach number of 5 the Reynolds number will be $\, \mathrm{Re} \,{=} 10^7$. The magnetic Reynolds number computed from the Spitzer resistivity formula at $T = 10^4 \, \mathrm{K}$ is even larger with $\, \mathrm{Rm} \,{=} 10^{21}$, yielding the magnetic Prandtl number $\, \mathrm{Pm} \,{=} 10^{14}$. derive a scaling for the small-scale dynamo growth rate in the case of large $\mathrm{Pm}$ and supersonic turbulence as $$\Gamma \propto \mathrm{Re}^{1/2}$$ meaning that we can extrapolate the growth rate we measured in our simulation to expected ISM Reynolds numbers obtaining the growth rate with a realistic galactic small-scale dynamo of $\Gamma = 1000 \, \mathrm{Gyr^{-1}}$ and an e-folding time of just $\tau = 1 \, \mathrm{Myr}$ respectively.

At the time of saturation, the magnetic pressure to turbulent pressure ratio is between 2.6% in the lower resolution case and 5% for the higher resolution. All of this is in agreement with the theoretical results of who predict the same three-phase evolution and give expected saturation levels. Although those numbers apply only in the limiting cases of very small or very large magnetic Prandtl numbers, they are still consistent with our result lying in between due to $\, \mathrm{Pm} \sim 1$ from both Reynolds numbers being caused numerical scheme. The saturation values are also in agreement with prior numerical studies, such as who find 2% with $\, \mathrm{Pm} = 2$ and solenoidal forcing at Mach 10 and who find 3% for $\, \mathrm{Pm} = 2$ and 5% for $\, \mathrm{Pm} = 5$ (both at Mach 11). The same holds for who report saturation ratios between 2% and 4% and also point out a trend of an increased ratio in the case of higher resolutions. The agreement of our results with the simulations mentioned above is remarkable considering that our numerical setup differs considerably.

With the magnetic field amplified up to full saturation, albeit not yet in equipartition, we have used that configuration to observe its evolution into a quiescent disk by turning off supernova feedback, thereby removing the main driver of turbulence. We find that that with the decay of turbulence, the gas cools down and falls onto a thin clumpy disk with dense arms. Inside these arms, we observe the magnetic field to be locally at equipartition with turbulence and thermal pressure, with field strengths of several $\, \mathrm{\mu G}$. Field lines inside the quiescent disk are aligned with the disk plane, with a strong toroidal component and slightly weaker radial component. These results are consistent with observations of spiral galaxies which usually have ordered magnetic field strengths of $5 \, \mathrm{\mu G}$ where the magnetic energy density is at equipartition with the turbulent energy density . The toroidal as well as the radial components are symmetric across the midplane, with an average pitch angle of $\langle p_{\mathrm{B}} \rangle = -12\, \mathrm{^\circ}$. This symmetry confirms measurements of the Galactic magnetic field and pitch angle measurements of various nearby galaxies . Furthermore, the magnetic field reversals along the radial distance from the centre of the galaxy confirm observations of nearby spiral galaxies where such reversals are found, usually attributed to their spiral structure.

# Conclusions

We have performed numerical experiments with an idealised setup of a dwarf galaxy to study the evolution of a galactic magnetic seed field as small-scale dynamo amplification occurs due to turbulence driven by feedback processes, such as supernova explosions. We have shown that, with the formation and the death of the first massive stars, the gas swiftly becomes turbulent and the initially weak seed field grows exponentially. The e-folding time of this dynamo process becomes shorter and shorter as the numerical resolution increases. Since the Reynolds number in our simulations is several orders of magnitude lower than that of a typical ISM plasma, we extrapolate the small-scale dynamo efficiency to an e-folding time of $\tau = 1 \, \mathrm{Myr}$. We conclude that even an initially very weak field strength of $B = 10^{-20} \, \mathrm{G}$ would be amplified up to dynamo saturation in a time span of only $30 \, \mathrm{Myr}$. Thus, a newly born turbulent galaxy can be highly efficient in amplifying its early seed fields, whether they be primordial or generated during structure formation, extremely rapidly. This mechanism would establish considerably strong fields since the early stages of the Universe in turbulent galactic environments, which would explain the high magnetic field strengths found by even at high redshift. The strong but sub-equipartition-strength field can then be further transformed by other processes like the $\alpha-\Omega$ dynamo. Moreover, the magnetic field will follow and even influence the history of its host galaxy through decisive events such as starbursts, mergers or more quiet phases. Therefore, magnetic fields should not be neglected when dealing with problems of galaxy evolution.

# Acknowledgements

This work was funded by the Swiss National Science Foundation SNF. All simulations were run on the Piz Dora cluster at the Swiss National Supercomputing Centre CSCS in Lugano, Switzerland.

[^1]: Contact e-mail: [firstname.lastname@example.com](email@example.com)