abstract: Recent observations have gathered a considerable sample of high redshift galaxy candidates and determined the evolution of their luminosity function (LF). To interpret these findings, we use cosmological SPH simulations including, in addition to standard physical processes, a detailed treatment of the Pop III$-$Pop II transition in early objects. The simulated high-$z$ galaxies match remarkably well the amplitude and slope of the observed LF in the redshift range $5 < z < 10$. The LF shifts towards fainter luminosities with increasing redshift, while its faint-end slope keeps an almost constant value, $\alpha \approx -2$. The stellar populations of high-$z$ galaxies have ages of 100-300 (40-130) Myr at $z=5$ ($z=7-8$), implying an early ($z>9.4$) start of their star formation activity; the specific star formation rate is almost independent of galactic stellar mass. These objects are enriched rapidly with metals and galaxies identified by HST/WFC3 ($M_{UV}<-18$) show metallicities $\approx 0.1 Z_{\odot}$ even at $z=7-8$. Most of the simulated galaxies at $z\approx 7$ (noticeably the smallest ones) are virtually dust-free, and none of them has an extinction larger than $E(B-V) = 0.01$. The bulk (50%) of the ionizing photons is produced by objects populating the faint-end of the LF ($M_{UV} < -16$), which JWST will resolve up to $z=7.3$. PopIII stars continue to form essentially at all redshifts; however, at $z=6$ ($z=10$) the contribution of Pop III stars to the total galactic luminosity is always less than 5% for $M_{UV}<-17$ ($M_{UV}<-16$). The typical high-$z$ galaxies closely resemble the GRB host galaxy population observed at lower redshifts, strongly encouraging the use of GRBs to detect the first galaxies.
author: Ruben Salvaterra$^1$, Andrea Ferrara$^2$ & Pratika Dayal$^3$
$^{1}$ Dipartimento di Fisica e Matematica, Universitá dell'Insubria, Via Valleggio 7, 22100 Como, Italy
$^{2}$ Scuola Normale Superiore, Piazza dei Cavalieri 11, 56126 Pisa, Italy
$^{3}$ SISSA/International School for Advanced Studies, Via Beirut 2-4 Trieste, Italy, 34014
title: Simulating high-redshift galaxies

# Introduction

The search for the most distant galaxies, located at the beginning of the cosmic dawn, is now entering its maturity. The last few years have witnessed a tremendous increase in the data available, and the number of candidates at redshifts as high as $z=10$, corresponding to only half a billion years after the Big Bang. This has been made possible by a combination of new technologies and refined selection methods. In the first class of triggers, it is easy to acknowledge the role of the Hubble Space Telescope (HST). Thanks to dedicated surveys including the Hubble Ultra Deep Field and its predecessor, the Hubble Deep Field, we have been able to collect information on the luminosities and number counts of galaxies located at the end of the reionization epoch. Immediately after, follow-up experiments performed with the newly installed Wide Field Camera (WFC3), yielding sky images in the F105W (Y-band, $1.05 \mu$m), F125W (J-band, $1.25 \mu$m) and F160W bands (H-band, $1.60 \mu$m), have allowed to push the exploration to very faint (e.g. AB mag = 28.8 in the above bands) galaxies as remote as $z=10$. In addition, the WFC3 crafted filters have considerably alleviated the contamination problem due to interlopers and provided more precise photometric redshift estimates. The standard selection method applied to these survey data sets is based on the dropout technique introduced by Steidel et al. (1996) and later constantly refined and improved by several authors (e.g. Giavalisco et al. 2004, Bouwens et al. 2007). Though this method has proved to be very solid in identifying high-redshift sources, it has the drawback that the exact source redshift cannot be determined with complete confidence. This uncertainty can be partly overcome by also using the longer wavelength infrared data, such as that provided by the *Spitzer* satellite; by building a more complete Spectral Energy Distribution (SED), the stellar mass, age and redshift of a given source can be constrained further. Other complementary techniques to search for distant ($z > 5$) galaxies are also widely used, among which the narrow-band spectroscopy (Malhotra et al. 2005; Shimasaku et al. 2006; Taniguchi et al. 2005; Kashikawa et al. 2006) aimed at detecting the Ly$\alpha$ line, carrying a large fraction of the bolometric luminosity, is definitely the most established one. Such narrow-band searches have yielded the record-holding most distant galaxy at $z=6.96$ (Iye et al. 2006). Finally, another series of experiments involve searching for remote galaxies behind foreground galaxy clusters acting as magnification lenses (Schaerer & Pelló 2005; Richard et al. 2008; Bradley et al. 2008). Although these searches result in deeper magnitudes, their interpretation is hampered by the lens modelling and by the extremely narrow field of views, rendering it difficult to keep cosmic variance under control. As a final remark, we note that the most distant, spectroscopically confirmed, cosmic object is a Gamma Ray Burst (GRB090423 at $z=8.2$, Salvaterra et al 2009b; Tanvir et al. 2009). Although not a galaxy, the presence of this indicator implies that star formation was already well under way at those early epochs, thus further encouraging deeper galaxy searches. In addition, the GRB can be seen as a signpost of the underlying galaxy which could possibly be detected in the future knowing its exact position. Such a finding would be of the utmost importance as GRBs are mostly associated with star forming dwarf galaxies (Savaglio, Glazebrook & Le Borgne 2009) which are now considered to be the dominant sources of (re)ionizing photons at high redshifts (Choudhury & Ferrara 2007; Choudhury, Ferrara & Gallerani 2008).

What have we learned from this wealth of experimental results ? The most solid piece of information that can be determined from the data appears to be the luminosity function (LF) and, less robustly, its evolution. It is useful to briefly recap the present observational situation marching towards increasing redshift. Bouwens et al (2007) present a comprehensive view of galaxy candidates from the UDF/ACS/GOODS fields using NICMOS in the redshift range $z=4-6$. They identify 1416 (627) V-dropouts ($i$-dropouts) corresponding to $z\approx 5$ ($z\approx 6$) down to an absolute UV magnitude of $M_{UV}\approx -17$ with a LF described by a Schechter function with characteristic luminosity and faint-end slope given by $M_{UV}^* = -20.64 \pm 0.13$ and $\alpha=-1.66\pm 0.09$ ($M_{UV}^* = -20.24 \pm 0.19, \alpha=-1.74\pm 0.16$) respectively. The same group (Bouwens et al. 2008) has extended the data analysis to include $z\approx 7$ $z$-dropouts (8 candidates at $z=7.3$) and J-dropouts (no candidates at $z \approx 9$). More recently, the installation of WFC3 on board the HST has triggered a new series of searches. Oesch et al. (2010) used data collected during the first-epoch WFC3/IR program (60 orbits) in the Y, J, H bands reaching a magnitude limit of AB$\approx 29 (5\sigma)$. They identify 16 $z$-dropouts in the redshift range $z=6.5-7.5$ from which they obtained a LF with ($M_{UV}^* = -19.91 \pm 0.09, \alpha=-1.77\pm 0.20$), essentially confirming the previous findings while extending it to fainter luminosities ($M_{UV} \approx -18$). Bouwens et al. (2010a) pushed the investigation to $z=8.0-8.5$ by using 5 Y-dropouts. Finally, Bouwens et al. (2009) were able to identify three J-dropouts. If confirmed, these sources would be the most distant objects detected so far. Similar studies using the same data has been performed by Bunker et al. (2010), who find a comparable number of $z-$ and $Y-$dropouts. McLure et al. (2010) did not apply specific color cuts as in the previous works, thus finding a larger number of candidates; however, they pointed out that about 75% of the candidates at $z>6.3$ (100% at $z>7.5$) allow a $z<2$ interloper solution. A recent analysis of the three HUDF and of the deep ($\sim 27.5$ AB mag), wide-area ($\sim 40$ arcmin$^2$) WFC3 Early Release Science reveals 66 and 47 candidate galaxies at $z=7$ and $z=8$, respectively (Bouwens et al. 2010b). After carefully modelling the selection volume of each field and of the possible contamination by spurious sources, the LF obtained from these data, while consistent with previous derivations of $M_{UV}^*$ and of the normalization, shows a steeper faint-end slope with $\alpha=-1.94\pm 0.24$ and $\alpha=-2.00\pm 0.33$ at $z=7$ and $z=8$, respectively.

Besides the LF, tentative information on the physical properties of these sources can be extracted from their SED, exploiting available *Spitzer* data (Eyles et al. 2005; Yan et al. 2006; Stark et al. 2009). In a recent study Labbé et al (2010b), based on follow-up *Spitzer/IRAC* observations, analyzed the SED of 12 $z$-dropout and 4 Y-dropout candidates. None of them is detected in the *Spitzer/IRAC* 3.6 $\mu$m band to a magnitude limit of AB=26.9 $(2\sigma)$, but a stacking analysis reveals a robust detection for the $z$-dropout sample and a strong upper limit for the Y-dropout one. The stacked SEDs are consistent with a stellar mass of about $10^9 M_\odot$, no dust reddening, sub-solar metallicity, and best-fit ages of about 300 Myr, implying a formation epoch $z\approx 10$. These results for the stacked sample should be compared with those obtained by Finkelstein et al (2010) who performed an object-by-object analysis and found similar ages but with a considerable spread, allowing ages as low as a few Myr.

One of the major triggers to look for very high-$z$ galaxies is the quest for the reionization sources. The ionizing photon budget provided by the candidate high-$z$ galaxies is often estimated by extrapolating their LF to lower luminosities, a step that introduces a considerable uncertainty in the final determination. Having this in mind, it is still interesting to note that most studies tend to agree on the fact that the integrated UV specific luminosity for the detected galaxies at $z=7-8$ falls short of accounting for the ionizing power required to reionize the intergalactic medium. Of course, this conclusion is subject to at least two major unknown factors, these being the gas clumping factor (affecting its ability to recombine), and the escape fraction of ionizing photons (affected by dust and neutral hydrogen absorption within galaxies). Additionally, the effect of poorly constrained ages and metallicities (including the presence of metal-free, massive Pop III stars), further complicate the calculation.

In spite of the large experimental effort, surprisingly little attention has been devoted by modelers to the very high redshift universe. Most of the work has so far concentrated on a semi-analytical approach (Stiavelli, Fall & Panagia 2004; Schneider et al. 2006; Bolton & Haehnelt 2007; Mao et al. 2007; Samui, Subramanian, Srianand 2009; Trenti et al. 2010) to compute the luminosity function, number counts and emissivity evolution of high-$z$ galaxies. Albeit quite fast and versatile, these methods cannot provide detailed information on the properties of the galaxies, often being based on simplified assumptions. Numerical dedicated simulations attempting to model galaxy populations beyond $z=5-6$ are also very scarce, with the partial exceptions constituted by the works by Nagamine et al. (2006) and Finlator, Davé & Oppenheimer (2007).

Our approach is novel and different in spirit from all the previous theoretical ones. As our main aim is to model very high redshift reionization sources, we can afford smaller simulation boxes, thereby reaching the high resolutions required to resolve the dominant reionization sources - dwarf galaxies. Most importantly, though, we have implemented a careful treatment of metal enrichment and of the transition from Pop III to Pop II stars, along with a careful modelling of supernova feedback. Here we are interested in deriving the LF plus other observables from the simulations and to cast them in a form that can be compared directly with the available data or used to make new predictions for the James Webb Space Telescope (JWST).

# Numerical simulations

For the present study we have performed a set of cosmological[^1] simulations using the publicly available code GADGET[^2] (Springel 2005) with an improved treatment of chemical enrichment as detailed in Tornatore, Ferrara & Schneider (2007, TFS07). A unique feature of the computation concerns the Initial Mass Function (IMF) of stars, which is taken to be different for Pop III and Pop II stars and depends on gas metallicity, $Z$. In brief, if $Z < Z_{\rm cr}$, the adopted IMF is a Salpeter law with lower (upper) limit of $100M_\odot$ ($500 M_\odot$) and only pair-instability supernovae ($140 M_\odot < M < 260 M_\odot$) contribute to metal enrichment (Heger & Woosley 2002). If $Z \ge Z_{\rm cr}$, the above limits are shifted to $0.1 M_\odot$ ($100 M_\odot$), respectively; stars above $40 M_\odot$ end their lives as black holes swallowing their metals. According to the canonical choice, we fix $Z_{\rm
  cr} = 10^{-4}Z_\odot$. These two populations, to which we will refer to as Pop III and Pop II stars respectively, differ also for their metal yield and explosion energy. Complete details of the simulation can be found in TFS07. The simulation follows the production and transport of six different metal species, namely: C, O, Mg, Si, S, Fe, based on which the locally appropriate IMF is selected. The simulated volume has a linear (comoving) size $L=10 h^{-1}$ Mpc with $N_p=2\times 256^3$ (dark+baryonic) particles, corresponding to a dark matter (baryonic) particle mass of $M_p=3.62 \times 10^6 h^{-1} M_\odot$ ($6.83
\times 10^5 h^{-1} M_\odot$); the corresponding force resolution is $2~$kpc. Our resolution does not allow us to track the formation of mini-halos, whose stellar contribution remains very uncertain due to radiative feedback effects (Haiman & Bryan 2006; Susa & Umemura 2006; Ahn & Shapiro 2007, Salvadori & Ferrara 2009). The computation is initialized at $z=99$ and carried on until $z=2.5$. Star formation is treated as a stochastic process, following Springel & Hernquist (2003) Basically, at a given time the star formation rate of a (multi–phase) gas particle is computed using a Schmidt-type law $\dot m = xm/t_*$, where $xm$ is the mass of cold clouds providing the reservoir for star formation. Within the effective star formation model by Springel & Hernquist (2003), the star formation time-scale, is computed as $t_*(\rho) = t_*^0 (n/n_*)^{-1/2}$, where $t_*^0=1.5$ Gyr and the density threshold for star formation, $n_* = 0.1$ cm$^{-3}$, are free parameters of the model chosen so as to reproduce the observed Kennicut relation within uncertainties.

Supernova winds are treated as in the original model by Springel & Hernquist (2003); however, for simplicity and because the mass load and kinetic energy fraction are unknown parameters, we have given the winds from both populations an initial velocity $v_w = 500$ km s$^{-1}$, which appears to be consistent with that derived from observations of high-z starburst galaxies. Wind particles are temporarily hydrodynamically decoupled until either (i) they have moved by a traveling length $\lambda = 2$ kpc, or (ii) their density has decreased below 0.5 times the star formation density threshold. The metals are donated by star particles to the surrounding gas ones using a SPH kernel as described in TFS07.

The gas photo-ionization and heating rates are calculated at equilibrium with a background ionizing radiation due to the combined contribution of galaxies and quasars, taken from Haardt & Madau (1996), shifted so that the intensity at 1 Ryd is $J_\nu= 0.3\times
10^{-21}$ erg  s$^{-1}$ Hz$^{-1}$, in agreement with Bolton et al. (2005). Gas cooling and wind treatment details are the same as in TFS07.

# Luminosity Function

As a first check of the simulation results we will compute the evolution of the LF of the simulated galaxies and compare it with available data. To this end we make a number of physical assumptions that are discussed in the following.

For each galaxy at a redshift $z$, the luminosity at rest-frame wavelength, $\lambda$, is the sum of the contribution of its Pop II and Pop III stars. The SED of Pop II stars is computed by running the Starburst99 code (Leitherer et al. 1999, Vazquez & Leitherer 2005) using the metallicities and stellar ages appropriate for the galaxy under consideration as obtained from the cosmological simulation. The SEDs of massive (metal-free) Pop III stars have been computed by Schaerer (2002) including the effect of nebular emission. For both populations the IMF is taken according to the same prescription used in the simulation and described in Sec. . The total luminosity of a galaxy is then given by

$$L_\lambda=L_{\lambda}^{\rm II}+L_{\lambda}^{\rm III}=
l_{\lambda}^{\rm II}(\tau^{\rm II},Z)\dot{M_*}^{\rm II}+l_{\lambda}^{\rm III}\dot{M_*}^{\rm III}\tau^{\rm III}$$

where $l_{\lambda}^{\rm II}(\tau^{\rm II},Z)$ is the SED template for Pop II stars with mean age $\tau^{\rm II}$ and metallicity $Z$ corresponding to a continuous star formation rate of 1 $M_{\odot}$ yr$^{-1}$; $\dot{M_*}^{\rm II}$ is the Pop II star formation rate. The SED template of Pop III stars, whose star formation rate is $\dot{M_*}^{\rm III}$, is $l_{\lambda}^{\rm III}$; $\tau^{\rm III} = 2.5\times 10^6$ yr is the mean lifetime of massive Pop III stars (Schaerer 2002). We have implicitly assumed that the emission properties of Pop III stars are constant during the short lifetime of these massive metal-free stars. Next, the luminosity is converted into absolute AB magnitudes, $M_{UV}$, where the suffix UV refers to the wavelength $\lambda= (1600, 1350, 1500, 1700, 1500, 1600)$ Å for $z=(5, 6, 7, 8, 9, 10)$, respectively. Though this wavelength choice is chosen to be consistent with the observational data we compare with, a different value would not affect the results in any sensible way, given the flatness of the spectrum in this short wavelength range.

```latex
\begin{figure*}
\begin{tabular}{lr}
\includegraphics[scale=0.32]{highzlf_fig2a.eps}& \includegraphics[scale=0.32]{highzlf_fig2b.eps}
\end{tabular}
\caption{Maps showing the distribution of galaxies with an observed flux larger than the JWST sensitivity limit of 1 nJy in the J-band (left panel) and H-band (right). The maps are a 2D cut of a 3D image produced by stacking simulation snapshots between $z \sim 7.6-11.6$. The vertical color bar gives the redshift of the object, the size of the galaxies scales with their flux in the range 1 nJy $< {\cal F} <$ 24.4 nJy.}
\label{maps}
\end{figure*}
```

The LF of galaxies at any redshift $z$ is obtained by counting the galaxies with a given absolute magnitude in each magnitude bin and dividing the final result by the total volume of the simulation and bin size (0.5 mag). We perform this procedure for the following six redshifts $z=(5, 6, 7, 8, 9, 10)$. The results are shown in Fig.  as solid histograms, where the error bars represent the Poisson error on the number of galaxies in each magnitude bin. These theoretical LFs are then compared to the experimental ones collected from the various analyses of the HUDF. For $z=10$ we show the upper limits on the LF obtained from the three available candidates identified by Bouwens et al. (2009).

Let us now analyze the results shown in Fig. in more detail. It is clear that the luminosity range sampled by the observations and our predictions is only partially overlapping. This is because on one hand, even the exquisite sensitivity of WFC3 is not sufficient to properly sample the faint-end of the LF ($M_{UV}\mathrel{\rlap{\lower 3pt\hbox{$\sim$}} \raise 2.0pt\hbox{$>$}}-18$); on the other hand, our simulations, which are specifically designed to properly resolve the very first galactic units in a relatively small volume, lack the most massive, rare objects which comprise the bright end of the LF. In spite of these shortcomings, we consider it a rewarding success that the amplitudes of the theoretical and experimental LFs match almost perfectly, and at the same time, have quite similar slopes at all redshifts for which data are available. This is even more striking as no attempts have been made to fit or adjust the theoretical curves to the observed LF, i.e. they have been computed directly from the simulation output with no free extra parameters.

Our results suggest two clear trends. First, the LFs shift towards fainter luminosities with increasing redshift, mimicking a pure luminosity or density evolution. This is quite consistent with the trend of an increasing $M^*_{UV}$ with redshift, found in the data by several groups (see the extended discussion in Ouchi et al. 2009), preferring a pure luminosity evolution. Second, the faint-end slope of the simulated LF does not vary (within errors) from $z=5$ to $z=10$, maintaining an almost constant value of $\alpha=-2.0$. This value is slightly larger than the one derived from the data (see Sec. ) at $z=5-6$ but perfectly consistent with the recent determination of $\alpha$ for the $z=7-8$ LF obtained by Bouwens et al. (2010b). This behavior is most likely produced by a combination of the halo mass function evolution, feedback effects and evolving stellar populations.

Clearly the faint-end slope will be better constrained by forthcoming facilities such as the JWST; the above successful test of our model allows us to make reliable predictions for surveys that will be performed with such instruments. For a deep exposure of $10^6$ s, JWST is expected to reach a photometric sensitivity of about $1$ nJy ($10\sigma$) allowing an investigation of the faint-end of the LF predicted by our simulations at least up to $z=9$ (see Fig. ). This will be particularly exciting because it will very likely unveil the physical properties of these objects which are now thought to be the main reionization sources, as we will discuss in detail in Sec. . We want to stress here that our simulation allow us to resolve properly (i.e. with more than 200 DM particles) the galaxy population that will be observed by JWST, being their total halo masses $\ge 10^9\;M_{\odot}$. This, toghether with the carefull threatment of the chemical feedback, makes our predictions particularly robust.

To give a visual idea of the high-$z$ ($z > 7.5$) galaxy population in a typical JWST field (of size 5.3 arcmin) we have produced the maps shown in Fig. which demonstrate the wealth of objects that JWST will be able to detect up to $z \mathrel{\rlap{\lower 3pt\hbox{$\sim$}} \raise 2.0pt\hbox{$>$}}11$. The maps are a 2D cut of a 3D image produced by stacking all the available simulation snapshots between $z \sim 7.6-11.6$, and filling in the gaps between snapshots by randomly rotating the orientation axes in all the three-dimensions simultaneously.

As it has emerged from the overview of the observational results given in the Introduction, dust seems to have only a minor effect on shaping the SED of faint high-redshift galaxies. This is easily understood, as we will see later on, as a result of the relatively low stellar masses and metallicities characterizing these objects. Dust can however become more important in luminous Lyman Break Galaxies (LBGs) and Lyman Alpha Emitters (LAEs) as many studies have shown (Lai et al. 2007; Atek et al. 2008; Nagamine, Zhang & Hernquist 2008; Dayal et al. 2008, 2009, 2010, 2011; Finkelstein et al. 2009).

The dust enrichment of galaxies can be followed self-consistently by post-processing our simulation through the model introduced by Dayal et al. (2010), using which, we can quantitatively check the previous statement. In brief, the model assumes that dust is produced by supernovae (we neglect here the contribution of AGB stars, though see Valiante et al. (2009) who found a significant contribution from these objects for population ages $>150-200$ Myr) and taking into account three processes: (a) dust forms in the expanding ejecta with a yield per SNII of $0.54 M_\odot$ (Todini & Ferrara 2001; Nozawa et al. 2003; Bianchi & Schneider 2007), (b) SNII destroy dust in the ISM they shock to velocities $> 100$ km s$^{-1}$, with an efficiency of 0.12 (McKee 1989), and (c) a homogeneous mixture of gas and dust is assimilated into star formation (astration). Once the dust mass is calculated for each galaxy in the simulation, we translate this into a value of $E(B-V)$ using the appropriate SN dust extinction curve given by Bianchi & Schneider (2007) as explained in Dayal et al. (2010). The resulting values of $E(B-V)$ for galaxies at $z\approx 7$ are shown in Fig. as a function of the stellar mass. Many galaxies, especially the smallest ones, are almost dust-free, and none of them shows a dust reddening value larger than $E(B-V)=0.009$. This evidence allows us to safely neglect the effects of dust on the UV LF.

# Properties of first galaxies

In addition to predicting the global evolution of the LF, a major strength of our study is that it makes possible to extract the physical properties of the high-$z$ galaxies which are a part of the faint-end of the LF. We start by concentrating on the stellar ages, reported in Fig. as a function of the UV magnitude, from which the following main conclusions can be drawn. On average, fainter (and less massive) objects tend to be younger at all redshifts, with typical ages for observable objects in the range 200-300 Myr at $z=5$ and 80-130 Myr at $z=7-8$, with the caveat that a considerable age spread exists at all luminosities. These ages imply that these galaxies started to form stars as early as $z=9.4$, clearly suggesting that their UV light might have influenced the cosmic reionization history. From a different perspective, we note that stellar ages of $\sim 100-150$ Myr are expected for galaxies at the sensitivity limit of WFC3 for $z=7-8$, in agreement with the observational estimates obtained by Labbé et al. (2010a) and Finkelstein et al. (2010).

A second important key physical parameter of pristine galaxies is their stellar and gas metallicity. In the following we will only discuss the stellar metallicity, keeping in mind that the two closely match each other. Somewhat surprisingly (but not unexpectedly) even the faintest galaxies appear to be already enriched to remarkable levels: at the JWST sensitivity threshold, we find $Z > 0.03 Z_\odot$ at all redshifts; galaxies identified by HST ($M_{UV} < -18$) systematically show metallicities in excess of $Z=1/10 Z_\odot$ even at $z=7-8$. Thus we come to the interesting conclusion that even at these early epochs, the self-enrichment, due to the metals produced, following the first star formation episodes is able to increase the metal abundances of such small objects to levels comparable to their present-day counterparts (e.g. the Magellanic Clouds). In addition, such high mean metallicites could in principle preclude the formation of Pop III stars according to the critical metallicity scenario (Schneider et al 2002, 2003; Schneider & Omukai 2010) which predicts $Z_{cr}=10^{-5\pm 1} Z_\odot$ as the upper limit for the formation of Pop III stars. However, as we will discuss in more detail later, inside these early structures, small pocket of (quasi) pristine gas may survive in which a relatively tiny amount of Pop III stars continue to form as pointed out by TFS07 and Jimenez & Haiman (2006). In brief, the scenario leads to the concept of a "PopIII wave", i.e. the physical phenomenon by which in each galaxy the formation of stars below the critical metallicity is progressively segregated towards the external regions of the galaxy, where almost unpolluted regions are still present. Until this process comes to an end (when metal pockets produced around the first stars forming regions reach a considerable volume filling factor in the system), PopIII and PopII formation modes coexist in the same galaxy. One might naively think that this process can take longer in large galaxies than in small ones, but it is (roughly) true only to zero-th order approximation, as in lower mass galaxies the efficiency of star formation is also depressed and the amount of time required, once scaled by the baryonic mass of the system, is not dramatically different from that of larger systems.

Taken together, Fig.  and provide a first guess of the properties of a typical high-$z$ galaxy with given absolute UV magnitude $M_{UV}$. This can be useful when comparing the observed photometric data with synthetic SEDs.

Fig.  shows the relation between the SFR and stellar mass of galaxies at $z=7$. Simulated objects follow an almost linear relation with significant spread, i.e. an almost constant specific star formation rate. This trend closely matches the one found in the analysis of stacked SEDs of WFC3 $z$-dropout by Labbé et al. (2010b), although a single object analysis reveals large errors in the determination of both the stellar mass and the SFR for these objects (Gonzalez et al. 2010). We note that simulated galaxies tend to have a slightly higher SFR (for a fixed stellar mass) than expected by the extrapolation of the relation to smaller objects. However, in a re-analysis of the observed $z=7$ sample including the effect of nebular continuum and line emission along with that of dust absorption, Schaerer & de Barros (2010) find smaller stellar masses and larger SFR with respect to previous works. Our simulated galaxies lie somewhat in between these two observational estimates.

Fig.  shows the evolution of the galaxy properties as a function of redshift for galaxies with different stellar masses. The mean age of stellar population for all galaxies in the simulation is found to decrease with redshift $\propto (1+z)^{-2}$; however, the age spread increases for less massive galaxies at any redshift. The average stellar metallicity is also growing with time for $M_*>10^7 M_\odot$, but show a much flatter (almost constant) evolution in the smallest objects, leveling at about 1/25 $Z_\odot$ for the tiniest star-forming galaxies. However, even for dwarf galaxies with $M_* \approx 10^5 M_\odot$ one can find individual objects enriched up to $1/10$ of solar metallicity already at $z=10$. This mass-dependent metallicity evolution is probably caused by the different ability to retain metals deposited by supernova explosions of the massive and dwarf populations. As the potential wells of the latter one are shallower, metals escape easily into the intergalactic medium thus setting an upper limit to the amount of metals than can be kept in their main body (Mac Low & Ferrara 1999). An obvious implication is that IGM metals preferentially come from small and common objects, thus resulting in a more homogeneous enrichment (Ferrara, Pettini & Shchekinov 2000; Madau, Ferrara & Rees 2001).

# Reionization sources

Having obtained the luminosity function and characterized some of the physical properties of the current high-$z$ candidate galaxies, we now turn to an analysis of their role in the reionization process.

The rate of ionizing photons from the $j$-th galaxy is given by

$$\dot{n}_{ion,j}=f_{esc}\left[Q^{\rm II}\dot{M}^{\rm II}_{j}+Q^{\rm III}\dot{M}^{\rm III}_{j}\tau^{\rm III}\right]$$

where $Q^{\rm II}$ ($Q^{\rm III}$) is the ionizing photon flux for Pop II (Pop III) stars, which of course is a function of the IMF, metallicity and stellar age of any given galaxy. Finally $f_{esc}$ is the escape fraction of ionizing photons. For simplicity, we assume a redshift-independent value of $f_{esc}$ for the two stellar populations. The ionization rate provided by galaxies in the simulation box per unit comoving volume at redshift $z$, $\dot{N}_{ion}(z)$, is then given by the sum over all galaxies at that redshift divided by the volume of the simulation.

The actual ionization rate must also include galaxies that are too rare to be caught in our relatively small simulation volume. To account for this correction, we add the ionization rate due to bright galaxies by integrating the observed LF (or the upper limits) from the luminosity of the brightest galaxy in the output down to very low magnitudes ($M_{UV}=-25$). The ionizing photon flux is obtained using a SED derived assuming that rare galaxies have the same stellar age and metallicity as the brightest simulated one. The extra contribution of the unaccounted bright-end of the LF is found to be at most $10$% of the total ionizing photon emission.

The evolution of the total specific ionization rate, $\dot{N}_{ion}$, is plotted in the top panel of Fig.  ($f_{esc}=0.2$) along with the same rate due to galaxies detected in the HUDF, $\dot{N}_{ion}^H$, or detectable by JWST, $\dot{N}_{ion}^J$. The ratios between ($\dot{N}_{ion}^H$, $\dot{N}_{ion}^J$) and $\dot{N}_{ion}$ are also shown for clarity in the bottom panel of the same Figure. HST is now resolving the sources that provide $\approx 1/3$ of the ionizing photon budget at $z=5$ and $\sim 20$% at $z=7-7.5$. This results is consistent with the limits set by the observed very steep faint-end slope of the $z=7-8$ LF (Bouwens et al. 2010b) and with the estimate of recent semi-analytical models (Choudhury et al. 2008, Trenti et al. 2010).

At the sensitivity limit of JWST, it will be possible to detect the bulk of ionizing sources up to $z\sim 7.3$, but at higher redshifts most of the ionizing photons will still be produced by sources that are too faint to be detected even by JWST.

The total ionization rate density $\dot{N}_{ion}(z)$ should then be compared with the recombination rate density of the IGM, $\dot{N}_{rec}(z)$, given by (e.g. Madau, Haardt & Rees 1999)

$$\dot{N}_{rec}=\frac{\langle n_H \rangle}{\langle t_{rec} \rangle}=10^{50.0}C_{HII}\left(\frac{1+z}{7}\right)^3 {\rm s}^{-1} {\rm Mpc}^{-3},$$

where $\langle n_H \rangle$ is the mean comoving hydrogen density in the Universe and $\langle t_{rec} \rangle$ is the volume-averaged recombination time for ionized hydrogen with an effective H$_{\rm II}$ clumping factor $C_{HII}=\langle n^2_{HII}\rangle/\langle n_{HII}\rangle^2$. The recombination rate density is shown in the top panel of Fig.  with dotted lines for different value of the clumping factor $C_{HII}$. For $f_{esc}=0.2$ and $C_{HII}=10$, the balance between ionization and recombination is obtained at $z\sim 6.8$. For $f_{esc}=0.1$ $\dot{N}_{ion}=\dot{N}_{rec}$ at $z=6$ assuming $C_{HII}=10$.

# Light from Pop III stars

Another piece of useful information than can be extracted from the simulation outputs is the relative fraction of normal (Pop II) and massive, metal-free (Pop III) stars. There are several questions to which we can provide quantitative answer: (i) do some of the current candidates contain Pop III stars ? (ii) in that case, what fraction of their UV luminosity is powered by them ? (iii) how is this fraction dependent on their $M_{UV}$ luminosity ? (iv) is there a clear observational signature imprinted by Pop III stars ?

The answer to the first question is straightforward: having analyzed the stellar populations of the simulated galaxies present at four observationally relevant redshifts, $z=(5, 6, 8, 10)$, we find that a fraction 0.07-0.19 (depending on $z$) of the galaxies contain at least some Pop III stars. We should not emphasize too much on the exact values of this Pop III/Pop II galaxy ratio as fluctuations in galaxy mass and star formation rate might introduce a very large dispersion. The most robust physical quantity to understand the relative importance of the two populations is the ratio of Pop III-to-Pop II star formation rates which is a decreasing function of time (see Fig. 1 of TFS07) never exceeding $10^{-3}$ below $z=10$.

More relevant is question (ii) above, whose answer can be obtained by inspecting Fig. , showing the ratio, $L_{UV,III}/L_{UV}$ of UV luminosities contributed by Pop III and Pop II stars as a function of $M_{UV}$. Light from Pop III stars becomes progressively more important towards fainter objects. This confirms previous findings (e.g. Schneider et al. 2006) that Pop III stars preferentially form in low-$\sigma$ peaks rather than in larger galaxies whose gas has already been enriched by several stellar generations. At the same time, we do not find galaxies containing only Pop III stars: this occurrence is made unlikely by the short lifetimes of such very massive stars. At $z=6$ ($z=10$) the contribution of Pop III stars to the total luminosity is always less than 5% for $M_{UV}<-17$ ($M_{UV}<-16$).

Among the candidate galaxies detected so far, we find that Pop III stars contribute less than a few percent to the total galaxy luminosity. Even at the detection limit of JWST, no Pop III-dominated galaxies ($\mathrel{\rlap{\lower 3pt\hbox{$\sim$}} \raise 2.0pt\hbox{$>$}}50$% of total luminosity) will be found, due to their extreme faintness ($M_{UV}\mathrel{\rlap{\lower 3pt\hbox{$\sim$}} \raise 2.0pt\hbox{$>$}}-14.$ at $z=6$). However, some objects having $\approx 10$% of their luminosity powered by Pop IIIs are present at $z=6$ at the 1 nJy sensitivity level reachable by the JWST.

Recently, many authors (Shapley et al. 2003, Nagao et al. 2008, di Serego Alighieri et al. 2008) have searched for the HeII 1640Å  emission line in the spectra of the so-called *dual emitters*, i.e. high-$z$ galaxies showing strong emission in both Ly$\alpha$ and HeII lines. The HeII line is usually taken as a well-defined signature of Pop III stars. Up to now, these searches have given negative results. Motivated by these attempts, we have computed the rest-frame equivalent width of the HeII line for the objects in our simulation box as $EW({\rm HeII})=L({\rm HeII})/L_{1460{\rm \AA}}$ where $L(\rm {HeII})=5.67\times 10^{-12} Q({\rm He}^+)$ is the luminosity of the HeII line in erg s$^{-1}$, $Q({\rm He}^+)$ is the rate of He ionizing photons, and $L_{1460{\rm \AA}}$ is the continuum luminosity at $\lambda=1460$Å  in erg s$^{-1}$ Å$^{-1}$. In particular, we focus here on $z=6$ which might be more easily accessible to present or future observations. Regretfully, the perspectives of direct detection of PopIII stars through this technique do not appear as very promising. For objects detectable in the JWST (HUDF) deep field survey, the expected HeII rest-frame equivalent width is $<0.5$ Å ($<0.1$ Å). Such small EWs will be very difficult to detect. The EW increases to more accessible values of about $10$ Å only if much fainter objects ($M_{UV} = -13$) could be observed. We have to underline that the above discussion implies that we cannot set limits on the total cosmic SFR in Pop III stars using the results of dual emitters searches given that the bulk of Pop III stars may be 'hidden' in galaxies much fainter than any present (and probably future) survey can detect (Scannapieco, Schneider & Ferrara 2003).

In conclusion, Pop III stars are found to form essentially at any redshift and in $< 20$% of the galaxies. However, their contribution to the total galaxy luminosity is very low, apart from the very faint objects and their detection could be extremely difficult even with the next generation of space telescopes.

# Early Gamma-Ray Bursts

Long Gamma-Ray Burst (GRBs) are powerful flashes of $\gamma$-rays that are observed with a frequency of about one per day over the whole sky. The $\gamma-$ray emission is accompanied by a long-lasting tail, called afterglow, usually detected over the whole electromagnetic spectrum. Their extreme brightness easily over-shines the luminosity of their host galaxy and makes them detectable up to extreme high redshifts, as shown by the discovery of GRB 090423 at $z=8.2$ (Salvaterra et al. 2009b, Tanvir et al. 2009). Metal absorption lines can often be identified in their afterglow spectra, allowing a study of the metal (and dust) content of the environment in which they blow. Finally, once the afterglow has faded, follow-up searches of the GRB host galaxy become possible. At low redshifts, GRBs are typically found in blue, low-metallicity dwarf galaxies with stellar masses $M_*\sim 10^{8-9}\;M_{\odot}$ and high specific star formation rates (Savaglio et al. 2009). These objects closely resemble the properties of high-$z$ galaxies identified in our simulations, whose mean specific star formation rates (SSFR) are $\approx 8-10$ Gyr$^{-1}$, albeit associated with a large spread (for $M_*\ge 10^8$, SSFR=1.5-7 Gyr$^{-1}$ at $z=6-8$). This suggests that high-$z$ GRBs can be used as signposts of the same faint galaxies that provide the bulk of the ionization photons in the early Universe[^3]

. Moreover, the study of their afterglows can provide new hints about the metal (and dust) content of the parent galaxy. Finally, it has been proposed that even Pop III stars may eventually blow as powerful GRBs (Fryer, Woosley & Heger 2001; Yoon, Langer & Norman 2006; Hirschi 2007; Komissarov & Barkov 2010). The observation of these Pop III-GRBs may provide a valuable way to detected the elusive, short living first stars. We will discuss these points in the following.

## GRBs as metallicity probes

Fig.  shows the mean stellar metallicity of the simulated sample of high-$z$ galaxies as a function of their absolute UV magnitude at different redshifts $z\ge 5$, i.e. the luminosity–metallicity relation for such objects. Although the experimental determination of such a relation would be of the utmost importance in testing theoretical predictions, in practice such effort is hampered by the extreme difficulty to infer $Z$ from lines in the galaxy spectrum with available facilities.

This problem can be considerably alleviated if a GRB could be found inside of one of these remote galaxies. In this case absorption features produced by heavy elements dispersed in the interstellar medium (ISM) surrounding the GRB would leave a characteristic and recognizable imprint on top of the GRB afterglow spectrum. Moreover, there is now a good agreement on that GRBs are usually hosted in relatively small, star-forming galaxies (Savaglio et al. 2009): these requirements would make the dwarf high-$z$ galaxies which, according to our findings, are the dominant population during the first cosmic billion year, optimal candidate hosts.

The only high-quality spectra burst at $z>6$ available so far, GBR 050904 at $z=6.3$ (Kawai et al. 2006), shows the expected metal absorption features, witnessing the presence of metals at the same redshift of the GRBs (Kawai et al. 2006). From this data, and with the further assumption that the measured sulfur (\[S/H\]=$-1.3\pm 0.3$) is a good proxy for metallicity, one can determine the metal content of the host galaxy. Berger et al. (2007) attempted a detection of the GRB host galaxy with HST and *Spitzer*, and were able to set an upper limit to the host luminosity of $M_{UV}\mathrel{\rlap{\lower 3pt\hbox{$\sim$}} \raise 2.0pt\hbox{$>$}}-20.3$, i.e. $M_{UV}\mathrel{\rlap{\lower 3pt\hbox{$\sim$}} \raise 2.0pt\hbox{$>$}}M_{UV}^*$ at $z=6$. By combining the HST and *Spitzer* upper limits, they set an upper limit to the stellar mass of the galaxy, $M_*\mathrel{\rlap{\lower 3pt\hbox{$\sim$}}\raise 2.0pt\hbox{$<$}}{\rm few}\times 10^9\;M_{\odot}$. This constraint is consistent with the predictions presented in Fig. (notice the data point corresponding to the Berger et al. 2007 determination), which might indicate that at $z\approx 6$ a galaxy with $Z\approx 0.05 Z_\odot$ is on average 2-3 magnitudes fainter than the detection limit of the experiment; in addition, such metal abundance level is quite typical for galaxies located at that epoch.

GRB 090423 (Salvaterra et al. 2009b, Tanvir et al. 2009) has been recently detected at $z=8.2$. The intrinsic properties of the burst (both of the prompt and afterglow phases) are similar to those observed at low/intermediate redshifts, suggesting that the progenitor and the medium in which the burst occurred are not markedly different from those of low-$z$ GRBs (Salvaterra et al. 2009b, Chandra et al. 2010). Chary et al. (2009) observed the field of GRB 090423 for 72 hours using *Spitzer*/IRAC at 3.6 $\mu$m, looking for its host galaxy. The observation was performed $\sim 46$ days after the GRB trigger, (corresponding to about 5 days in the burst rest-frame). A weak source was detected at the location of the GRB afterglow with $L_{AB}=27.2\pm 0.3$. The observed flux is consistent with the power-law decay of the GRB afterglow, suggesting that the source is still contaminated by the GRB emission and implying a limit on the absolute magnitude of the host of $M_{3900{\rm \AA}}>-19.96$. From Fig. 3, we expect that the host galaxy of GRB 090423 should be enriched at a level of a few percent solar. This relatively high metallicity may explain the high equivalent neutral hydrogen column density measured in the X-ray afterglow, although a wide range of systematic effects or the contamination by low-$z$ intervening absorption systems may be a more valuable, alternative explanation for the observed absorption (Chandra et al. 2010).

These examples nicely show how GRBs represent an unique tool to study the high-$z$ Universe. In particular, the observation of their optical-NIR afterglow may allow to eventually study the evolution of the mass-metallicity relation up to very high-$z$. To achieve this goal, high resolution and good signal-to-noise afterglow spectra are required as soon as possible after the GRB detection in $\gamma$-rays. As nicely demonstrated by GRB 090423, GRBs are easily detectable well beyond any other astrophysical object. Indeed, up to $\sim 5$% of all GRBs detected by the *Swift* satellite are expected to be at $z>6$ (Salvaterra & Chincarini 2007, Salvaterra et al. 2009b). Future missions (e.g. EXIST, XENIA, SVOM) will rapidly increase the high-$z$ GRB sample (Salvaterra et al. 2008) allowing a statistical study of GRB hosts in the high-$z$ Universe and a direct check of the galaxy metal enrichment history at those early epochs. In particular, EXIST, thanks to its 1.1m optical-NIR telescope, will be able to take the GRB afterglow spectrum only 300 s after the trigger, allowing an on-board direct measure of the redshift and the identification of metal absorption lines when the afterglow is still sufficiently bright even for high-$z$ bursts (Grindlay et al. 2009).

## GRBs from Pop III stars

It has been widely discussed (see Bromm & Loeb 2007 for a review) whether GRBs can arise from the collapse of massive, metal-free stars. While the large envelopes of these objects may suppress the emergence of relativistic jets out of their surface, the weak winds expected for low-metallicity stars can prevent angular momentum loss during their evolution, producing the rapidly rotating central configurations needed to produce a GRB. In the following we will assume that massive Pop III stars produce GRBs.

Excluding Pop III progenitors in the mass range \[140,260\] $M_{\odot}$, that will explode as Pair Instability Supernovae (PISN, Heger & Woosley 2002) leaving no remnant, we have two possible channels through which GRBs can occur: (i) 100-140 $M_{\odot}$ (Yoon et al. 2006, Hirschi 2007) and (ii) 260-500 $M_{\odot}$ (Fryer et al. 2001, Komissarov & Barkov 2010). In the latter case, since both the luminosity and the duration are thought to be proportional to the black hole mass, an extremely bright and long GRB is expected (Fryer et al. 2001), while more typical luminosities and durations may be expected in the case of smaller progenitors (Hirschi 2007).

We can estimate a strong upper limit for the rate of Pop III-GRB detections as follows. Suppose a fraction $f_{GRB}$ of all Pop III stars with masses in the ranges discussed above produce GRBs with a typical beaming angle $\theta$, and that all Pop III-GRBs are detectable by present-day satellites given the their extreme brightness. Then the observed rate of Pop III-GRB, $R_{GRB}(>z)$, is given by $$R_{GRB}(>z)=5.5\times 10^{-3} \left(\frac{\theta}{6^\circ}\right)^2 \int_z dz\, f_{GRB}\, \eta\,
\frac{\dot{\rho}^{III}(z^\prime)}{1+z^\prime} \frac{dV}{dz}$$ where $\dot{\rho}^{III}(z)$ is the total Pop III star formation rate at redshift $z$, $\eta=\int_{M_{min}}^{M_{max}} \phi(m_*) dm_*/\int_{100}^{500} m_* \phi(m_*) dm_*$ is the number of Pop III stars with masses in the range $[M_{min},M_{max}]$ given the assumed IMF $\phi$, and $V$ is the cosmological volume per unit solid angle. The factor $(1+z)^{-1}$ accounts for time dilation effects due to redshift. The computed rate of Pop III-GRB at $z>6$ is then $R_{GRB}\sim 7.5 (2.5) f_{GRB}  (\theta/6^\circ)^2$ yr$^{-1}$ sr$^{-1}$ for GRB progenitors in the mass range 100-140 $M_{\odot}$ (260-500 $M_{\odot}$). This rate is of the same order of that expected for normal (i.e. Pop II/PopI progenitors) GRBs (Salvaterra & Chincarini 2007, Salvaterra et al. 2009b). However, we stress here that very likely $f_{GRB}\ll 1$, resulting in lower detection rates. The detection of one of such Pop III-GRBs might represent the most promising way to directly detect the very first stars to have formed.

# Critical discussion

The present results allow to build a coherent and quantitative description of the properties of elusive high-redshift, possibly primordial, galaxies. They are also very useful to interpret the data coming from deep surveys as the HST/WFC3 and future ones. However, there is considerable room for improvement left by our study. In the following we would like to elaborate on the uncertainties and shortcomings of our findings.

We first note that resolving the dwarf galaxy population and following the PopIII transition process along with the large variety of physical processes implemented in the simulation limits the size of the cosmic volume that can be simulated. Resolution is certainly an important issue, as it is well known (see, e.g. Governato et al. 2010) to affect the simulated star formation rates and cause the loss of sub-galactic structures. The dependence of the results from resolution has been presented an analyzed for the same set-up of the present simulations in Tornatore et al. (2007) to which we refer the interested reader (see Fig. 1 of that paper). Resolution might also alter the details of the "Pop III wave" evolution, since galactic substructure allows star formation to occur at the edges of galaxies as well. The PopIII-PopII transition is also very dependent on the assumed IMF of PopIII stars. Our conclusions are valid under the assumption that PopIII stars were very massive ($M\ge 100\;M_{\odot}$) and the first metal production is driven by the explosion of pair-instability SNe (see Schneider et al. 2006 for alternatives).

Our box is also too small to properly describe cosmic reionization, let alone that we are not even attempting to properly treat radiative transfer. These issues have been already addressed in previous works of our group; as already stated we are concerned here with the properties of high-z galaxies, which are presumably more affected by their internal physics rather than by the environment, as we explain below. Comparing the simulated volumes with the observed ones is very challenging as considerable uncertainty exists on the latter (see discussion in Appendix B of Bouwens et al. 2009). However, as it could be induced from the extension of the LF towards the most luminous and rarest objects at z=7, we estimate that the volume sampled by experiments should be about 30 times larger than our simulated one.

The next caveat comes from the fact that we have neglected the effects of minihalos (virial temperature $< 10^4$ K). Our resolution does not allow us to track the formation of such objects, whose stellar contribution remains very uncertain due to radiative feedback effects (Haiman & Bryan 2006; Susa & Umemura 2006; Ahn & Shapiro 2007, Okamoto, Gao & Theuns 2008, Salvadori & Ferrara 2009). The presence of such small collapsed structures, if able to form stars, could alter the reionization history to some extent and increase the number counts of high-redshift galaxies, if detectable. As far as reionization is concerned, it has already been shown by Choudhury & Ferrara (2007) that acceptable fits to all relevant reionization data can be obtained without any need for PopIII stars. The bulk of the ionizing photons in those models is produced by halos with virial temperatures just above $10^4$ K, with increasingly better solution if normal, PopII stars are allowed to form in minihalos. As shown in Fig. , where we plot the UV magnitude as function of the total mass of z=7 galaxies, sources detectable with JWST have halo masses $M_h \mathrel{\rlap{\lower 3pt\hbox{$\sim$}} \raise 2.0pt\hbox{$>$}}10^9 M_{\odot}$. This corresponds to circular velocities $v_c \mathrel{\rlap{\lower 3pt\hbox{$\sim$}} \raise 2.0pt\hbox{$>$}}50$ km s$^{-1}$. These objects are large enough that suppression by UVB photoionization filtering is at best marginal, if not negligible at all, as most of the works above agree upon. Internal mechanical feedback might be indeed more important. This process is however is already included at best in the simulations when computing the star formation rate of individual galaxies, modulo the many uncertainties that still plague our understanding of such phenomenon.

# Summary

By using high resolution simulations specifically crafted to include the relevant physics of galaxy formation, along with a novel treatment of the metal dispersion that allows us to follow the PopIII-PopII transition as dictated by the critical metallicity scenario, we have been able to reproduce the observed UV LFs over a wide redshift range, $5 < z < 10$. We have also shown, by combining the simulation outputs with a dust model previously developed for LAEs (Dayal et al. 2010), that dust effects at $z\approx 7$ should be marginal, although this statement depends on many details (the dust properties and the dust distribution scale in the ISM) that are poorly constrained at this time. The general picture that can be drawn from our investigation is broadly consistent with the available data and therefore can be used to make specific predictions for the JWST. It is then useful to schematically summarize the main findings of the present work:

-   The simulated high-$z$ galaxy UV LFs match remarkably well with the amplitude and slope of the observed LFs in the redshift range $5 < z < 10$.

-   The LF shifts towards fainter luminosities with increasing redshift, mimicking a pure luminosity (or density) evolution. The faint-end slope of the LF does not vary from $z=5$ to $z=10$, keeping an almost constant slope value of $\alpha=-2$.

-   Many galaxies at $z\approx 7$, especially the smallest ones, are virtually dust-free, and none of them shows dust extinctions larger than $E(B-V)=0.009$. This evidence allows us to safely neglect the effects of dust on the UV LF.

-   The stellar population of high-$z$ galaxies shows typical ages in the range 100-300 Myr at $z=5$ and 40-130 Myr at $z=7-8$, implying that they started to form stars as early as $z=9.4$. These objects are enriched rapidly with metals and galaxies identified by HST/WFC3 show metallicities $\approx 1/10 Z_{\odot}$ even at $z=7-8$. The trend of decreasing metallicity (and increasing spread) towards low mass halos indicates that small galaxies are more affected by supernova feedback and loose a larger fraction of the heavy elements they produce.

-   The relation between the star formation rate and stellar mass of simulated follows an almost linear relation with significant spread towards the lowest masses, implying an almost constant specific star formation rate.

-   The bulk of the ionizing photons is produced by objects populating the faint-end of the LF. These galaxies are beyond the capabilities of current survey, but JWST will be able to resolve them up to $z=7.3$.

-   Massive PopIII stars continue to form essentially at all redshifts and in $<20$% of the galaxies. However, their contribution to the total galaxy luminosity is negligible ($<$ 5%) for all objects with the marginal exception of the extremely faint ones; their detection will be tremendously difficult even for the next generation of space telescopes.

-   The typical high-$z$ galaxies closely resemble the GRB host population observed at lower redshifts. This fact suggests that GRBs can be used to detect and study these objects, providing unique information about the first stages of structure formation. In particular, they can be used to extend the study of the mass-metallicity and its evolution to very high redshifts. Moreover, if PopIII stars end their lives in a GRB explosion, the detection of one of these bursts might represent the most promising way to directly detect the very first stars.

# Acknowledgments

We thank R. Schneider and L. Tornatore for collaborative support. Discussions and the stimulating environment at `DAVID IV`, held at OAArcetri, Florence is kindly acknowledged.

[^1]: Throughout the paper, we adopt a $\Lambda$CDM cosmological model with parameters $\Omega_M = 0.26$, $\Omega_{\Lambda} = 0.74$, $h=0.73$, $\Omega_b=0.041$, $n=1$ and $\sigma_8=0.8$, in agreement with the 3-yr WMAP results (Spergel et al. 2007).

[^2]: www.mpa-garching.mpg.de/galform/gadget/

[^3]: Note that GRBs may be biased tracers of the cosmic star formation rate (e.g. Daigne et al. 2006; Salvaterra & Chincarini 2007; Salvaterra et al. 2009a). However, if this is related to the existence of a metallicity threshold for GRB formation of $Z<0.3\;Z_{\odot}$ as required by the collapsar model (e.g. MacFadyen & Woosley 1999) then we expect that high-$z$ GRBs can be fair cosmic tracers.


abstract: The amazing advances being made in the fields of machine and deep learning are a highlight of the Big Data era for both enterprise and research communities. Modern applications require resources beyond a single node's ability to provide. However this is just a small part of the issues facing the overall data processing environment, which must also support a raft of data engineering for pre- and post-data processing, communication, and system integration. An important requirement of data analytics tools is to be able to easily integrate with existing frameworks in a multitude of languages, thereby increasing user productivity and efficiency. All this demands an efficient and highly distributed integrated approach for data processing, yet many of today's popular data analytics tools are unable to satisfy all these requirements at the same time.
  .
  In this paper we present *Cylon*, an open-source high performance distributed data processing library that can be seamlessly integrated with existing Big Data and AI/ML frameworks. It is developed with a flexible C++ core on top of a compact data structure and exposes language bindings to C++, Java, and Python. We discuss Cylon's architecture in detail, and reveal how it can be imported as a library to existing applications or operate as a standalone framework. Initial experiments show that Cylon enhances popular tools such as Apache Spark and Dask with major performance improvements for key operations and better component linkages. Finally, we show how its design enables Cylon to be used cross-platform with minimum overhead, which includes popular AI tools such as PyTorch, Tensorflow, and Jupyter notebooks.
author:

bibliography: bib/references.bib
title: High Performance Data Engineering Everywhere

# Introduction

Large-scale data processing/engineering has gone through remarkable transformations over the past few decades. Developing fast and efficient *Extract, Transform and Load* frameworks on commodity cloud hardware has taken center stage in handling the *information explosion and Big Data*. Subsequently, we have seen a wide adoption of Big Data frameworks from Apache Hadoop , Twister2, and Apache Spark to Apache Flink in both enterprise and research communities. Today, Artificial Intelligence (AI) and Machine Learning (ML) have further broadened the scope of data engineering, which imposes faster and more integrable frameworks that can operate on both specialized and commodity hardware.

One important question is whether those existing Big Data frameworks utilize the full potential of the computing power and parallelism available to process data. Both Big Data and AI/ML applications spend a goodly amount of time pre-processing data. Minimizing the pre-processing time clearly increases the throughput of these applications. Productivity is another important aspect of such frameworks. Most available data analytics tools are implemented using a rapid programming language such as Java, Python or R. This allows data engineers to develop applications without diverging into the details of complex distributed data processing algorithms. Still, we rarely see these two aspects (high performance and productivity) meet each other in the existing Big Data frameworks. We have also seen the world increasingly moving towards user-friendly frameworks such as NumPy, Python Pandas or Dask . Big Data frameworks have been trying to match this by providing similar APIs (for example, PySpark, Dask-Distributed). But this comes at the cost of performance owing to the overheads that arise from switching between runtimes.

We believe that a data processing framework focused on high performance and productivity would provide a more robust and efficient data engineering pipeline. In this paper we introduce *Cylon*: a high-performance, *MPI* (Message Passing Interface)-based distributed memory data parallel library for processing structured data. Cylon implements a set of relational operators to process data. While "Core Cylon" is implemented using system level C/C++, multiple language interfaces (Python and Java (R in future)) are provided to seamlessly integrate with existing applications, enabling both data and AI/ML engineers to invoke data processing operators in a familiar programming language.

Large-scale ETL operations most commonly involve mapping data to distributed relations and applying queries on them. There are distributed table APIs implemented on top of Big Data frameworks such as Apache Spark  and Apache Flink  which are predominantly based on Java programming language. Furthermore SQL interfaces are developed on top of these to enhance the usability. Initial use cases of Big Data frameworks were mostly based on text data (for example, analyzing web scrolls of the Internet/social media, time series data, etc.). But data engineering has ventured beyond text data to analyzing multi-dimensional data such as image, video, and text-to-speech. Further due to programming language barriers, integration between Big Data frameworks with AI/ML applications written in C++/Python and scientific applications are not as efficient as possible. There is also a discordance between these frameworks and high performance computing frameworks such as MPI implementations. The fact that these large-scale data processing operations exist as separate frameworks also limits their use cases due to the overhead in setting them up.

We envision data processing as a high performance library function that should be available everywhere, including deep learning, data processing frameworks, distributed databases and even services. This paper discusses how we achieve this using Cylon. Its flexible C++ core and language interfaces allow it to be imported as a library for applications or run as a standalone framework. It includes a table abstraction and a set of fundamental relational algebraic operations that are widely used in data processing systems, allowing Cylon to couple seamlessly with existing AI/ML and data engineering infrastructures. Internally it uses a compact Apache Arrow data format. The initial results show significant performance improvements compared to current state-of-the-art data processing frameworks. In Section , we elaborate on the functionality, architecture and features of Cylon. Section discusses the idea of "data processing everywhere" and how Cylon achieves it. Section illustrates how Cylon performs against popular data processing frameworks.

# Cylon

Cylon[^1] is a data engineering toolkit designed to work with AI/ML systems and integrate with data processing systems. This vision is highlighted in Figure  where Cylon is shown to support common data structures and systems. It can be deployed either as a library or a framework. Big Data systems like Apache Spark, Apache Flink and Twister2 can use Cylon to boost the performance in the ETL pipeline. For AI/ML systems like PyTorch, Tensorflow and MXNet, it acts as a library to enhance ETL performance. Additionally, Cylon is being expanded to perform as a generic framework for supporting ETL and efficient distributed modeling of AI/ML workloads.

Cylon currently provides a set of distributed data-parallel operators to extract, transform and load structured relational data. These operators are exposed as APIs in multiple programming languages (e.g., C++, Python, Java) that are commonly used in Machine Learning and Artificial Intelligence platforms, enabling tight integration with them. When an operator is invoked in any of these platforms, that invocation is delegated to the "Core Cylon" framework, which implements the actual logic to perform the operation in a distributed setting. A high level overview of Cylon, along with its core framework, is depicted in Figure .

Cylon core has a table abstraction to represent structured data. When a table is created in a distributed context, each worker or process will hold a partition of the data that logically belongs to the table. However each process can work on their own portion of the table, as if they are working on the entire dataset. Cylon "local operators" are executed on the local data and distributed operators depend on these local operators after distributing data as per operator requirements. Distributed operators are implemented based on the Bulk Synchronous Parallel (BSP) approach and the framework synchronizes local operations as needed. In order to take the complexity of distributed programming away from the user, Cylon internally performs network-level operations and abstracts out the distributed nature of the operators. Those network operations function on top of a layer where communication can take place over either TCP, Infiniband or any other protocol supported by the underlying communication layer of the framework. In the following subsections we discuss each of these layers in detail.

## Data Model

Traditionally, data processing systems are divided into two categories: (1) Online Transaction Processing (OLTP) and (2) Online Analytical Processing (OLAP). OLTP usually deals with a large number of atomic operations such as inserts, updates, and deletes, while OLAP is focused on bulk data processing (bulk reads and writes) with low volume of transactions.

Cylon workloads predominantly fall under the OLAP category. The data layer of Cylon is based on Apache Arrow. "Arrow Columnar Format" provides the foundation for Cylon Table API. This allows seamless integration of other open source frameworks such as Spark and libraries such as Pandas, Parquet and NumPy. Additionally, Arrow also provides zero copy reads, which drastically reduces the overhead of switching between language runtimes.

Further, columnar data formats have a number of advantages when used for OLAP. Predominantly, each column will exist contiguously on storage (memory or disk), and will be homogeneously typed. This increases the performance by allowing SIMD operations, utilizing caches more efficiently due to the data locality while also allowing effective compression of data. This was the basis for the Apache Parquet columnar data format developed for Hadoop ecosystem. Apache Arrow extends this strategy to an in-memory language-agnostic data structure.

## Operators

On top of the table abstraction, Cylon implements five fundamental operations that can be used to create a complete data processing system. Each operator has two modes of execution: *local* and *distributed*.

*Local operators* do not use the communication layer. Instead they work entirely on the data available and accessible locally to the process, although these operators are capable of performing computations on the data in volatile memory as well as on the data in the disk. The performance of these operations is bound usually by IO and CPU capacity. Hence, local operators are optimized to utilize disk-level and memory-level caches efficiently. Some of the local operations we have implemented in the initial version of the Cylon are Join, HashPartition, Union, Sort, Merge, and Project.

*Distributed operators* use the network layer at one or multiple points during the operator's life-cycle (beginning, middle, or end). In other words, a distributed operator is one or more local operators coupled with one or more network operators. Network operators can be considered as I/O and network bound operators that involve minimum amount of CPU based on the underlying protocol. Initially we have implemented the "All to All" network operator which is widely required when implementing the distributed counterparts of the local operators. Figure shows how the distributed join operation has been composed by combining two local operators (HashPartition, Local Join) and one network operator (AllToAll).The performance of distributed operators depends on I/O, CPU, and network performances. Current operators implemented in Cylon are listed in Table , though this list is expected to grow.

### Select

Select is an operation that can be applied on a table to filter out a set of rows based on the values of all or a subset of columns. When Select is called in a distributed environment, Cylon applies the predicate function provided by the user on the locally available partition of the distributed table. This operation is a pleasingly parallel one where network communication is not required at all.

### Project

Cylon does not enforce any limits on the amount of columns or rows that a table can have. As long as the hardware resources permit, it can handle a dataset of any complexity. Project can be used to create a simpler view of an existing table by dropping one or more columns. Project is considered the counterpart of Select, which works on columns instead of rows. Similar to Select, Project is also a pleasingly parallel operation, and Cylon applies it on distributed partitions of the table without having to perform any synchronization over the network.

### Join

Join operation can be used to combine two tables based on the values of a common column. Cylon implements four types of Join operations;

1.  Inner Join : Includes records that have matching values in both tables.

2.  Left (Outer) Join : Includes all records from the left table and just the matching records from the right table.

3.  Right (Outer) Join : Includes all records from the right table and just the matching records from the left table.

4.  Full Outer Join : Includes all records, but combines the left and right records when there is a match.

We have implemented two different algorithms to perform the above four operations.

1.  Sort Join : Sorts both tables based on the join column and scans both sorted relations from top to bottom while merging matching records to create the joined table.

2.  Hash Join : Hashes the join column of one relation (preferably the smallest relation), and keeps the hashes in a hash map. Scans through the second relation while hashing the join column to find the matching records from the first table's hash map.

Although Join is a straightforward operation when performed locally, all matching records need to be in the same process when performed in a distributed context. Therefore Cylon couples Join operation with a shuffling phase to redistribute data based on the Join column value. We use a hash-based partitioning technique where the records with the same Join column hash will be sent to a designated worker/process. At the end of the shuffling phase, local Join can be applied on the re-partitioned table.

### Union

The Union operation can be applied on two homogeneous tables (those having similar schema) to create a single table which includes all the records from both source tables with duplicates removed. Similar to Join, in order to perform Union on a distributed dataset, all similar records need to be in the same process. Cylon performs shuffling as the initial step of the distributed Union operation. Unlike with Join, Union considers all the columns (properties) of a record when finding duplicates. For that reason, the hash value of the entire record (row) is considered when performing the hash - partitioning.

### Intersect

When applied on two homogeneous tables the Intersect operation produces a table with similar rows from both. Cylon couples this operation with a shuffling phase similar to the Union operator's distributed implementation.

### Difference

This operation can be considered as the opposite of Intersect. When the Difference is applied on two homogeneous table, it produces the final table by adding all the records from both tables but removing all similar records. Since the similar records need to be identified, this operation has also been coupled with a shuffling phase.

```latex
\begin{table*}[t]\centering
{\renewcommand{\arraystretch}{1.3}
\begin{tabular}{|l|p{14cm}|}
\hline
Operator   & Description                                                                \\ \hline
Select     & Select operator works on a single table to produce another table by selecting a set of attributes that matches a predicate function that works on individual records.                                                 \\ \hline
Project    & Project operator works on a single table to produce another table by selecting a subset of columns of the original table.                    \\ \hline
Join       & Join operator takes two tables and a set of join columns as inputs to produce another table. The join columns should be identical in both tables. There are four types of joins with different semantics: inner join, left join, right join and full outer join. \\ \hline
Union      & Union operator works on two tables with an equal number of columns and identical types to produce another table. The produced table will be a combination of the input tables with duplicate records removed.                                 \\ \hline
Intersect  & Intersect operator works on two tables with an equal number of columns and identical types to produce another table that holds only the similar rows from the source tables.                                       \\ \hline
Difference & Difference operator works on two tables with an equal number of columns and identical types to produce another table that holds only the dissimilar rows from both tables.                                            \\ \hline
\end{tabular}
}
\caption{Cylon operations\label{tb:ops}}
\end{table*}
```

## Communication Layer

Although the communication layer of Cylon was initially developed based on OpenMPI, that implementation can be easily replaced with a different one such as UCX. This will enhance Cylon's compatibility to run on a wide variety of hardware devices that have different native capabilities, including GPUs, and different processor architectures such as ARM and PowerPC. Transport layer options will also be widened with different communication layer implementations.

A Cylon application can utilize multiple communication layer implementations within the same process by defining multiple Cylon Contexts, which is the API layer abstraction of the underlying stack of communication and transport layers.

Data analytic systems either use an event-driven model, where data producers and consumers are decoupled, or synchronous models where producers and consumers work together at the same time. Classic parallel computing with Message Passing Interface (MPI) uses the latter approach where senders synchronize with the receivers for transferring messages. Big Data systems blend these two , with batch systems using the event-driven model and streaming systems using the synchronous model.

The producers and consumers are decoupled in time in an event-driven model. The data generated by a producer can be consumed in a later time suitable for the consumer, and not the time imposed by the producer. This allows greater flexibility in designing applications with distributed parts that can work independently.

Cylon uses synchronized producers and consumers for transferring messages. In contrast, Apache Spark employs an event-driven model for communication between its tasks.

## Transport Layer

At the time of writing, Cylon has the capability to communicate using any transport layer protocol supported by OpenMPI, including TCP and Infiniband. Additionally, all the tuning parameters of OpenMPI are applicable for Cylon since the initial implementation is entirely written based on the OpenMPI framework.

# Data processing Everywhere

One of the main goals of Cylon is to be a versatile library which facilitates data processing as a function (DPAF) and thus provide efficient data engineering across different systems. When working over multiple systems, data representation and conversion is a key factor affecting performance and interoperability. Cylon internally uses Apache Arrow data structure, which is supported by many other frameworks such as Apache Spark, TensorFlow, and PyTorch. Apache Arrow can be converted into other popular data structures such as NumPy and Pandas efficiently. In addition our core data structures can work with zero copy across languages. For example, when Cylon creates a table in CPP, it is available to the Python or Java interface without need for data copying.

Cylon C++ kernels efficiently support data loading and data processing. These functions can be used either in distributed or local setting. Most of the deep learning libraries like PyTorch, Tensorflow and MXNet are designed on top of such high performance kernels. Cylon APIs are made available to the user in a similar manner. Such designs lead to lower frictions in system integration. With these design principles, we envision the following scenarios where Cylon could work with other systems to create rich applications.

1.  Data processing as a library

2.  Data processing as a framework

3.  Accelerating existing data processing frameworks

## Data Processing Library

Cylon can be directly imported as a library to an application written in another framework. In a Python program, this integration is a simple module import. Cylon Python API currently supports Google Colab with an experimental version and supports Jupyter Notebooks with fully-fledged compatibility. In the model prototyping stage, setting up additional configurations and installation details become a bottleneck to researchers. Having smooth integration makes this process much easier.

A sample program snippet is shown in  where Cylon is imported to Pytorch application. Cylon can act as a library to load data efficiently by using either Arrow or Cylon data loaders. The Table API can then take over for data pre-processing. After the data pre-processing stage, the data can be converted to Pandas and then to Tensors in the AI framework. For data loading, Cylon Python API currently provides support for PyTorch distributed data loaders. This minimizes the effort of integrating Cylon Python APIs and PyTorch data loading. After this stage, the usual data loader object can be used to extract the tensors and run the training program.

Figure shows the current positioning of Cylon in deep learning integration. To further enhance the distributed operations, we can add specific support to deep learning settings such as NCCL .

## Data Processing Framework

Cylon can also perform as a separate standalone distributed framework to process data. As a distributed framework, Cylon should bring up the processes in different cluster management environments such as Kubernetes, slurm and Yarn. After this it accesses the core library to process the data. Cylon has a distributed backend abstraction to plug in various cluster process management systems. At the moment it works as a standalone framework with the MPI backend. Connections to cluster environments such as Kubernetes can be added through the resource management functions of existing frameworks like Twister2  or Dask .

## Accelerating existing data processing frameworks

There are two main areas in which Cylon can accelerate existing data frameworks. In terms of non-deep learning workloads, Cylon can be used as a plugin to provide an efficient communication API which encapsulates the state-of-the-art communication libraries such as MPI, NCCL, and UCX. In addition to this, Cylon can act as a high performance kernel for any Big Data system to perform relational algebra functions through the Table API. For JVM-oriented distributed data processing engines like Apache Spark, Apache Flink or Apache Storm, Cylon could provide a valuable addition with the Java and Python APIs. We are currently integrating Cylon kernels with the Twister2:TSet abstraction to provide efficient data pre-processing.

## Integrating with workflow systems

Any of the above three modes of Cylon can be used with workflow systems. Here workflow systems refers to continuously integrated applications. Data pre-processing, training, inference, post-analytics are linked in workflows. In modern use cases, data-driven action triggering has become more relevant. Support of standard file formats and in-memory data formats are important when transferring data between various parts of workflows.

# Experiments

We analyzed the strong and weak scaling performance of Cylon for the following operators and compared their performance against an existing well-known Big Data framework, Apache Spark.

1.  Join - Joins are a common use case of columnar-based traversal. Cylon implements two join algorithms: Hash join and Sort join. Results are presented for both algorithms.

2.  Union - Unions without duplicates; a use case for row-based traversal.

Additionally we compared Cylon performance against Dask + Dask-Distributed Python library, which is a popular data science library. We present the results of Dask side-by-side with Cylon and Spark.

We also analyzed the overhead of using Cylon on Java and Python environments. This provides an indication of how well Cylon can be integrated with Deep Learning and AI frameworks such as PyTorch, Tensorflow and MXNet while minimizing ETL processing time. In the following section we will present our experiment setup and results in detail.

## Setup

The tests were carried out in a cluster with 10 nodes. Each node is equipped with Intel^^ Xeon^^ Platinum 8160 processors. A node has a total RAM of 255GB and mounted SSDs were used for data loading. Nodes are connected via Infiniband with 40Gbps bandwidth.

Cylon was built using g++ (GCC) 8.2.0 with OpenMPI 4.0.3 as the distributed runtime. *Mpirun* was mapped by nodes and bound sockets. Infiniband was enabled for MPI. For each experiment, 16 cores from each node were used, totaling 160 cores.

Apache Spark 2.4.6 (hadoop2.7) pre-built binary was chosen for this experiment. Apache Hadoop/ HDFS 2.10.0 acted as the distributed file system for Spark, with all data nodes mounted on SSDs. Both Hadoop and Spark clusters shared the same 10-node cluster. To match MPI setup, *SPARK_WORKER_CORES* was set to 16 and *spark.executor.cores* was set to 1.

Dask and Dask-Distributed 2.19.0 was used with Pip installation. Dask Distributed cluster was used, in the same nodes as mentioned previously, with *nthreads=1* and varying *nprocs* based on the parallelism. All workers were equally distributed among the nodes.

For each test case, CSV files were generated with 4 columns (1 int_64 as index and 3 doubles). The same files were then uploaded to HDFS for the Spark setup and output counts were checked against each other to verify the accuracy. Timings were recorded only for the corresponding operation (no data loading time considered).

## Scalability

### Weak Scaling

```latex
\begin{figure*}[]\centering
\begin{tabular}[width=\textwidth]{|c|c|}
\hline
Join (Inner) & Union (Distinct)
\\
\hline
%
  {\small\begin{tabular}[t]{@{}c@{}}
  \includegraphics[width=70mm]{images/weak_j.png}\\(a)
  \end{tabular}}%

&
%
  {\small\begin{tabular}[t]{@{}c@{}}
  \includegraphics[width=70mm]{images/weak_u.png}\\(b)
  \end{tabular}}%

\\
\hline
\end{tabular}
\caption{Cylon weak scaling (Log-Log plots) [H = Hash-based, S = Sort-based]}
\label{fig:perf}
\end{figure*}
```

For the weak scaling, tests were carried out for parallelism from 1 to 160 while allocating 2 million rows per core per relation (left/ right). Hence the first case would have a total work of 2 million rows while the last case would have 320 million rows. The results are depicted in Figure .

Figure (a) shows Inner-Join results and (b) shows Union (Distinct) results. Both Join algorithms and Union implementation of Cylon shows sound scalability behaviour. As such both can be seen showing a flat line in the log-log plot for Cylon, indicating that the framework behaves as expected when adding more nodes with similar work. As the number of workers increases, the time for the completion grows owing to the increased communication among workers.

Compared to Cylon Joins, Union behaves poorly in a higher number of nodes. A possible reason for this is the row-based traversal of the table, which could nullify the advantages of a columnar data format.

Without loss of generality, we have plotted timings of Spark for the same experiments. In both cases Cylon perform better than Spark in terms of wall clock time for the operations, while both have similar upward trending curves at a higher number of workers.

### Strong Scaling

For the strong scaling, tests were carried out for parallelism from 1 to 160 while keeping total work at 200 million rows per relation (left/ right).The results are shown in Figure . It demonstrates individual speed-up over its own sequential test (ex: Cylon hash join speed-up over its sequential time)

```latex
\begin{figure*}[]\centering
\begin{tabular}[width=\textwidth]{|c|c|}
\hline
 Join (Inner) & Union (Distinct)
\\
\hline
%
  {\small\begin{tabular}[t]{@{}c@{}}
  \includegraphics[width=80mm]{images/strong_j1.png}\\(a)
  \end{tabular}}%

&
%
  {\small\begin{tabular}[t]{@{}c@{}}
  \includegraphics[width=80mm]{images/strong_u1.png}\\(b)
  \end{tabular}}%

\\
\hline
\end{tabular}
\caption{Cylon strong scaling (Log-Log plots) [H = Hash-based, S = Sort-based]}
\label{fig:sperf}
\end{figure*}
```

Figure (a) shows Inner-Join results and (b) shows Union (Distinct) results. As the number of workers increases, the work per core reduces. Both Join and Union speed-up is expected to follow a linear trend in the log-log plot. Cylon plots confirm this expectation. When the number of workers increases, the speed-up reaches a plateau.

## Cylon, Spark vs. Dask

```latex
\begin{tabular}{|c|c|c|c|c|c|}
\hline
\multirow{2}{*}{\textbf{Workers}} & \multirow{2}{*}{\textbf{\begin{tabular}[c]{@{}c@{}}Dask\\ Time(s)\end{tabular}}} & \multirow{2}{*}{\textbf{\begin{tabular}[c]{@{}c@{}}Spark \\ Time(s)\end{tabular}}} & \multicolumn{3}{c|}{\textbf{Cylon}}                   \\ \cline{4-6}
                                  &                                                                                &                                                                                & \textbf{Time (s)} & \textbf{v. Dask} & \textbf{v. Spark} \\ \hline
1                                 & -                                                                                & 586.5                                                                              & 141.5             & -                & 4.1x              \\ \hline
2                                 & -                                                                                & 332.8                                                                              & 116.2             & -                & 2.9x              \\ \hline
4                                 & 246.7                                                                            & 207.1                                                                              & 56.5              & 4.4x             & 3.7x              \\ \hline
8                                 & 134.6                                                                            & 119.0                                                                              & 27.4              & 4.9x             & 4.3x              \\ \hline
16                                & 134.2                                                                            & 62.3                                                                               & 13.2              & 10.1x            & 4.7x              \\ \hline
32                                & 113.1                                                                            & 39.6                                                                               & 7.0               & 16.1x            & 5.6x              \\ \hline
64                                & 109.0                                                                            & 22.2                                                                               & 4.0               & 27.4x            & 5.6x              \\ \hline
128                               & 70.6                                                                             & 18.1                                                                               & 2.5               & 28.1x            & 7.2x              \\ \hline
160                               & 68.9                                                                             & 18.0                                                                               & 2.3               & 30.0x            & 7.8x              \\ \hline
\end{tabular}
```
```latex
\begin{figure*}[t]\centering
\begin{tabular}[width=\textwidth]{|c|c|}
\hline
 Join (Inner) & Union (Distinct)
\\
\hline
%
  {\small\begin{tabular}[t]{@{}c@{}}
  \includegraphics[width=80mm]{images/compare.png}\\(a)
  \end{tabular}}%

&
%
  {\small\begin{tabular}[t]{@{}c@{}}
  \includegraphics[width=80mm]{images/strong_jall.png}\\(b)
  \end{tabular}}%

\\
\hline
\end{tabular}
\caption{Cylon, Spark vs. Dask strong scaling (Log-Log plots) [H = Hash-based, S = Sort-based]}
\label{fig:cylon-compare}
\end{figure*}
```

Figure (a) shows a strong scaling wall-clock time comparison between Cylon, Spark and Dask. The same strong scaling setup for Inner-Joins was used in this comparison. When comparing with Dask and Spark, Cylon performs better than them on the wall-clock time. For this 200 million line join, it scales better than both of the other frameworks. It should be noted that Dask failed to complete for the world sizes 1 and 2, even when doubling the resources. It continued to fail even with the factory *LocalCluster* settings, with higher memory.

Cylon shows better strong scaling, reaching a higher individual speedup. As shown in Table for a single worker (serial) Inner-joins, Cylon Hash, Cylon Sort, and Spark took 141s, 164s and 587s respectively. For Union, Cylon and Spark took 34s and 75s respectively. Thus not only does Cylon show better scaling, it achieves a superior wall-clock speed up because its serial case wall-clock time is an improvement on Spark.

Figure (b) shows the results for the Union (Distinct) operation. Unfortunately Dask (as of its latest release) does not have a direct API for distributed Union operation. As a result the comparison is limited to Spark and Cylon. As the graph depicts, Cylon performs better than Spark, with more than 2x better performance at each experiment.

## Overhead between C++, Python & Java

Figure shows the time taken for Inner-Join (Sort) for 200 million rows while changing the number of workers. It seems clear that the overheads between Cylon and its Cython Python bindings and JNI Java bindings are negligible.

# Related Work

Databases and Structured Query Language (SQL) lay at the heart of ETL pipelines until the emergence of Apache Hadoop . Viewed as the first generation of Big Data analytics, Hadoop provides a distributed file system (HDFS) and a data processing framework based on the MapReduce programming model . Even though Hadoop scales into terabytes of data over commodity hardware clusters, having a limited functional API, as well as high overheads due to disk-based operations and difficulty in managing applications, made it less attractive for the impending ETL applications.

Apache Spark succeeded Hadoop with the introduction of Resilient Distributed Dataset (RDD) abstraction. Built on the Scala, Spark provided a fast distributed in-memory data processing framework, with built-in SQL capability, streaming analytics and Python API. In later versions Spark developers enhanced the performance by optimizing queries and added more user-friendly table-like APIs that mimic Python Pandas *DataFrames*. Having to communicate between Python and Java run-times greatly affects PySpark's performance. An extension was introduced that uses Apache Arrow in-memory columnar data format in the Python environment, which resolves this issue to a certain extent. Unfortunately this enhancement is not available for the Java/Scala APIs. Spark initially proposed a machine learning library, MLlib, but with the wide adoption of specialized AI/ML and deep learning frameworks, it is now used purely as an ETL framework for AI/ML data manipulation. Apache Flink is another popular data analytics tool that emerged after Apache Spark, which was also developed in Java. Flink is geared for real-time streaming analytics and native iterative processing. It also provides a table-like API on both Python and Java environments. Both Spark and Flink use Py4J as the intermediary to communicate between Java and Python runtimes in a trade-off of performance for usability.

Pandas is the most popular library used in the data science community today, providing a rich toolkit for ETL operations. It is built on top of the Numpy library. *DataFrame* is the main table-like data structure of Pandas. The concept of *DataFrames* has inspired many other frameworks, as mentioned previously. The main drawback of Pandas is its lack of a distributed data abstraction as well as distributed operations. Since 2016, Dask has offered a distributed DataFrame abstraction like Python Pandas. Using distributed communication primitives, Dask provides standard Pandas operations like group-by, join, and time series computations. Since both Pandas and Dask DataFrames have been developed in Python, they suffer the inherent inefficiencies of the language environment. The application of DataFrames for large-scale ETL pipelines is still not very popular, even though they provide a user-friendly programming interface.

GPU hardware is becoming very popular for deep learning workloads. GPUs have many SIMD threads that are connected to high-bandwidth memory. CuDF integrates GPU computing capabilities to Pandas like DataFrame abstraction that can be used for ETL pipelines. It also integrates with Dask to enabe distributed computations on GPU DataFrames. It is built on top of the Apache Arrow columnar data format. Modin is a similar GPU-based dataframe library inspired by Pandas API. It uses Ray or Dask for distributed execution. With increasing GPU memory and hardware availability, it is inevitable that they would also be used for ETL workloads, which would complement deep learning workloads in turn.

Apache Spark is one of the pioneer dataflow systems to support deep learning workloads. BigDL framework developed on top of the Apache Spark ecosystem is one of the prominent research examples done in integrating deep learning with Apache Spark. Additionally, Horovod is another distributed deep learning framework which initially supported TensorFlow. It also uses Apache Spark as a backend for data pre-processing. In supporting the existing deep learning frameworks, there has been a recent contribution from another platform called Ray. Ray is an actor-based distributed system developed on top of a Python-backend. The major advantage of Ray is its ease of use as a regular Python library to bridge the gap between classical Big Data systems.

# Conclusions & Future work

In this paper we described the Cylon data processing library. We showcased its data model, how it can inter-operate between different systems efficiently, as well as its core operations and performance. We saw significant gains in efficiency compared to existing systems, proving our assumption that there is much room for improvement. Furthermore we discussed how Cylon fits into the overall data engineering of an application.

We are working on extending the Cylon operations to use external storage such as disks for larger tables that do not fit into memory. For disk-based operations, a dataflow graph-based API is more suitable due to the streaming nature of computations. At the initial stage we have implemented the fundamental relational algebraic operations. We are planning to add more operations to enhance the usability of the system.

# Acknowledgement

This work is partially supported by the National Science Foundation (NSF) through awards CIF21 DIBBS 1443054, nanoBIO 1720625, CINES 1835598 and Global Pervasive Computational Epidemiology 1918626. We thank Intel for their use of the Juliet and Victor systems, and extend our gratitude to the FutureSystems team for their support with the infrastructure.

[^1]: https://github.com/cylondata/cylon