author: K. T. Chyży; M. Soida; D.J. Bomans; B. Vollmer; Ch. Balkowski; R. Beck; M. Urbanik
date: Received date/ Accepted date
subtitle: A galactic wind in a ram pressure wind
title: Large-scale magnetized outflows from the Virgo Cluster spiral NGC 4569

# Introduction

NGC 4569 is a bright spiral galaxy (Sb) whose projected angular distance to the Virgo Cluster center (M87) is only $1.7^{\circ} = 0.5$ Mpc[^1]. Because of it brightness and large diameter ($D_{25}=9.5' = 47$ kpc) Stauffer et al. () have questioned its cluster membership. NGC 4569 has about one tenth the Hi content of a field galaxy of the same morphological type and the same size (Giovanelli & Haynes ). It shows a strongly truncated Hi disk (Cayatte et al. ), most probably a signature of strong stripping by the intracluster medium which pervades the Virgo cluster (Cayatte et al. ). Tschöke et al. () did not find any soft X-ray emission (0.1–0.4 keV) from the northern half of the disk, whereas it is pronounced in the southern disk (cf. Fig ). These findings are consistent with a ram pressure scenario where the galaxy is moving to the north-east through the intracluster medium. The H$\alpha$ emission distribution is, as the Hi, sharply truncated at 30% of the optical radius (Koopmann et al. ). In addition, an anomalous Hi and H$\alpha$ arm is detected to the west of the galactic disk. Simulated Hi gas distributions and velocity fields of a more edge-on ram pressure stripping event (the inclination angle between the disk and the orbital plane is $35^{\circ}$), where the maximum ram pressure occurred $\sim 300$ Myr ago, are consistent with the Hi observations (Vollmer et al. ).

Tschöke et al. () discovered a diffuse extraplanar region of X-ray and H$\alpha$ emission to the west of the galactic disk. This was the first evidence of a direct connection between the hot X-ray gas and the H$\alpha$ emission at scales of 10 kpc. This ionized gas is flowing out from the disk at a velocity of 120 km s$^{-1}$ (Bomans et al. ). The most probable source of this outflow is a central starburst (Barth & Shields , Tschöke et al. ). Present-day AGN activity is ruled out due to the lack of a compact point source in the ASCA hard band (Tschöke et al. ), a missing radio continuum point source (Neff & Hutchings , Hummel et al. ), and spectral synthesis analysis (Barth & Shields ). Gabel & Bruhweiler () dated the nuclear starburst in the inner $\sim 30$ pc to $5-6$ Myr based on optical and UV HST spectra. In addition, Keel () found a more extended ($\sim 300$ pc) population of A-type supergiants whose age is greater than $\sim 15$ Myr. Thus the nucleus of NGC 4569 contains at least two distinct young stellar populations: a very young UV core ($5-6$ Myr) and a spatially more extended region dominated by A supergiants.

In this work we use radio polarimetry to study the gas outflows from NGC 4569. Total power emission and, first of all, polarized radio continuum emission represent a very sensitive tracer of flows of diffuse magnetized gas, often unnoticed in the Hi and H$\alpha$ lines or in X-rays (Beck et al. , Soida et al. ). While the polarization allows to study the magnetic field in the plane of the sky, the Faraday rotation measures the sign and strength of the magnetic fields along the line-of-sight. The knowledge of the spectral index variations across radio continuum structures allows to trace the history of relativistic electron population transported with the gas and magnetic fields. In the absence of an in-situ acceleration mechanism the relativistic electrons lose their energy via synchrotron emission. Since electrons with higher energies lose their energy more rapidly, the radio spectrum steepens with time (electron aging). To obtain maps of Faraday rotation measures and spectral index we performed observations at two frequencies: 4.85 GHz and 8.35 GHz. Additionally we used the Hi continuum at 1.4 GHz obtained by Vollmer et al. (). The observations and the data reduction are presented in Sect. . The results are shown in Sect.  and discussed in Sect. . Finally, we give our conclusions in Sect. .

# Observations and data reduction

Observations at 8.35 GHz and with 1.1 GHz receiver bandwith were made using the single-horn receiver in the secondary focus of the 100-m Effelsberg radio telescope[^2]. The galaxy has been observed by making the maps (called coverages), scanned alternatively in R.A. and Dec. A total of 22 coverages was obtained. Four data channels were recorded. The first two channels contain total power signals. The correlations of the left- and right-handed circular polarization signals (giving Stokes Q and U) are recorded in the other two channels. At 4.85 GHz we used the two-horn system in the secondary focus of the Effelsberg telescope with the receiver of 0.5 GHz bandwith. Four data channels (two Stokes I as well as Q and U – see above) were recorded for each horn. We obtained 10 azimuth-elevation coverages of NGC 4569.

The telescope pointing was checked at time intervals of about 2 hours by making cross-scans of nearby strong point sources. The flux density scale was calibrated by mapping the highly polarized source 3C286 and computing its total power flux densities of 4.47 Jy at 8.35 GHz and 7.44 Jy at 4.85 GHz using the formulae of Baars et al. (). The same calibration factors were used for total power and polarized intensity, which yields a mean degree of polarization of 3C286 of 11% at 8.35 GHz and 10.5% at 4.85 GHz, in agreement with published values (Tabara & Inoue ).

The data reduction was performed using the NOD2 data reduction package (Haslam ). At 8.35 GHz all the total power coverages were combined into the final Stokes I map using the spatial frequency weighting method (Emerson & Gräve ). At 4.85 GHz we combined the total power information from both horns, using the "software beam-switching" technique (Morsi & Reich ). This was followed by a restoration of total intensities (Emerson et al. ), map transformation to R.A./Dec. coordinates and spatial frequency-weighted combination leading to the final Stokes I map.

Because of azimuthal mounting of the radio telescope the Q and U data at both frequencies were corrected for the rotation of the telescope reference frame of polarization with respect to the sky. At 8.35 GHz the distributions of Stokes parameters were combined into final Q and U maps using the same technique as for total power data. At 4.85 GHz the Stokes U and Q data for each coverage from both horns were averaged, then rotated to the R.A./Dec. frame and combined into final Q and U maps. A digital filtering process, which removes spatial frequencies corresponding to noisy structures smaller than the telescope beamwidth, was applied to final maps of all the Stokes parameters at both frequencies. The Q and U maps were finally converted into maps of polarized intensity, accounted for the "positive noise bias" in the way described by Wardle & Kronberg (). We also computed the distributions of *apparent* (i.e. uncorrected for Faraday rotation – small in our case) polarization B-vectors, defined as $\mathrm{arctan(U/Q)}+90\degr$.

Instrumental polarization of the Effelsberg radio telescope forms specific "butterfly patterns" in Q and U maps at the level of about 1% of an unpolarized signal (see Klein et al. for examples). They yield an almost axisymmetric structure in polarized intensity and in orientation of polarization vectors, constant in the reference frame of the telescope. Corrections for the azimuthal mounting makes the vectors of instrumental polarization rotated by the parallactic angle. In case of observations at a wide range of parallactic angles (which is our case) the superposition of spurious polarization patterns with various angles reduces the influence of instrumental polarization by factor of several. With the unpolarized intensities discussed in further Sections any instrumental polarization is in our case well below the noise.

The r.m.s. noise levels in total and polarized intensity at 8.35 GHz are 0.25 mJy/b.a. and 0.045 mJy/b.a., respectively. At 4.85 GHz the corresponding noise levels are 0.4 mJy/b.a. in total intensity and 0.09 mJy/b.a. in polarized intensity.

At 1.4 GHz we used the continuum residuals after subtracting the Hi line signal from the observations by Vollmer et al. (). Because of a low sensitivity (bandwidth of only 3 MHz) a convolution to the beamwidth of $65\arcsec\times 50\arcsec$ was applied to show the low-surface brightness structure. No polarization information was recorded at this frequency. The r.m.s. noise level in this map is 0.3 mJy/b.a.

# Results

## The global distribution of radio emission at 4.85 GHz

In contrast to the X-ray emission (Tschöke et al. ) the total power brightness at 4.85 GHz shows large extensions on both sides of the galactic disk (Fig. ). These large structures were totally unexpected. In the eastern lobe the radio surface brightness initially decreases shortly with distance up to about 6($\sim 30$ kpc) and then quickly drops. The western lobe is coincident with the X-ray emission and shows a more gradual decrease in surface brightness. Its extent is smaller than the eastern lobe (4$\sim 20$ kpc). At the position of the dwarf galaxy IC 3583 bright total power emission is seen.

The disk of NGC 4569 and both lobes show significant polarization at 4.85 GHz (Fig. , ). In the disk the B-vectors (i.e. E-vectors rotated by 90) are parallel to the disk plane, but in the extended radio structures the B-vectors are mostly perpendicular to the disk. The mean polarization degree in the disk is 5% increasing to 15% in the extraplanar radio-emitting regions. The radio emission from IC 3583 is polarized as well, with a degree of polarization of about 12%. The B-vectors are oriented towards NGC 4569.

The polarized intensity in the eastern lobe forms a structure elongated perpendicularly to the disk, only barely resolved along its minor axis (Fig. ). In contrast, the polarized emission on the western side extends smoothly along the whole disk edge. Even taking into account the beam-smeared contribution from IC 3583 leaves the polarization considerably more extended along the disk on the western than on the eastern disk side.

## Details of the radio structure at 8.35 GHz and 1.4 GHz

Our maps at 8.35 GHz with an angular resolution of 1 show the details of the total power and polarization structure of NGC 4569 (Fig. ). The eastern lobe appears to be a detached region with some hints of a radio bridge connecting it to the disk. Only its northern part is polarized by some 10% with B-vectors almost perpendicular to the disk. The western lobe has a similar shape in total power. However, it is somewhat weaker at 8.35 GHz than the eastern region. On this image IC 3583 is separated from NGC 4569.

Our observations at 8.35 GHz show that the polarized emission on the western and NW disk side is concentrated in three separated features (Fig. ): the most northern polarized peak is associated with IC 3583 and the ridge of polarized emission seen at 4.85 GHz west of the disk appears to be composed of two regions at the distance of about 3  from the centre of NGC 4569. The fact that they apparently coincide with regions showing little total power emission is due to a five times higher noise level in total power channels (due to confusion with background sources and an unpolarized atmospheric noise). We can thus detect significant polarization even where the total power signal is weak and very noisy. Both polarization peaks west of the disk show B-vectors perpendicular to the disk plane.

The map at 1.4 GHz made from the line-free channels of the Hi observations by Vollmer et al. () has the highest angular resolution of our data, but contains no polarization channels. The eastern lobe seems again to be a detached structure, a thin radio bridge connecting it to the disk of NGC 4569 is also confirmed (Fig. ). The western extension has a more complex structure and seems to be connected to the disk. In its southern part a relatively narrow ridge of total power emission extends away from the disk. It coincides roughly with the southern polarized extension seen west of the disk at 8.35 GHz (Fig. ). Bright radio emission from IC 3583, which might be connected to that of NGC 4569 via another putative faint radio bridge, is also visible in Fig.  and . The possible interactions between the galaxies has already been discussed by Tschöke et al. () mentioning the faint southern H$\alpha$ spur, seen in the H$\alpha+$NII image of IC 3583, pointing towards NGC 4569. This is confirmed by a new Fabry Perot H$\alpha$ image and velocity field obtained for these galaxies (Chemin et al. ).

# Discussion

## Spectral index map

Our results clearly indicate the presence of an unexpected bipolar outflow from the inner disk of NGC 4569. To determine the history of relativistic electrons in the extended lobes we computed the distribution of the spectral index $\alpha$ ($S_\nu \propto \nu ^{\alpha}$) between 8.35 GHz and 4.85 GHz over the whole source structure (Fig. ). The total power maps at both frequencies were convolved to a common beamwidth of 2. The disk has a rather steep spectrum with a slope of about $-1.0$. The diffuse western lobe has an even steeper spectrum with a slope steeper than $-1.3\, \div\,-1.4$. The steepening of the spectral index is due to the energy loss of the relativistic electrons via synchrotron emission. Since electrons with higher energies lose their energy more rapidly, the spectrum steepens if no re-acceleration of the electrons takes place. Thus the observed steepening of the spectral index in the western lobe is due to the aging of the relativistic electrons. Apparent flattenings at the most extreme western edge and at the northern disk boundary are found in regions of a very weak signal and may be artifacts of local background variations (a deviation of 1.5 r.m.s value in the 8.35 GHz map may explain the observed flattening). Acceleration of the cosmic rays in the southern part of the western lobe, with slightly flatter spectral index, cannot be excluded, but high resolution radio data would be needed to clarify this.

On the other hand, the region of flat spectrum ($\sim-0.6$) in the eastern lobe is certainly real. The slope of about $-0.6$ is only slightly steeper than the injection spectrum of relativistic electrons in a strong non-magnetic shock in a non-relativistic gas with compression ratio of 4 (Beck & Krause ). Since no tracers of ionized gas are found in the existing H$\alpha$ data in this region, an increased thermal fraction is unlikely, thus in-situ electron acceleration in a large-scale shock must take place.

At 8.35 GHz only 70% of the total flux density comes from the disk, the remaining 30% being emitted by the extended structures. Similar fractions are found at 1.49 GHz. Thus, a substantial amount of magnetic and cosmic ray energy is expelled from the galaxy to the intergalactic space. Such vertical nuclear outflows are known to exist in some disk galaxies with strong nuclear starbursts or AGN activity, like M82 (Reuter et al. ), the Circinus Galaxy (Elmouttie et al. ) or NGC 4258 (Krause et al. ). All these mentioned objects have extremely powerful central sources, dominating their radio emission. In NGC 4569 the data at 1.49 GHz processed with the highest possible resolution (beamwidth of 37 $\times$ 20= 3 kpc $\times$ 1.6 kpc) show that the unresolved central source comprises no more than 15 mJy, thus only some 10% of the total flux density. High resolution 6 cm and 20 cm data revealed a flux density of 1.5 mJy at 6 cm and 10 mJy at 20 cm in the inner $4'' = 330$ pc (Neff & Hutchings , Hummel et al. ). According to Gabel & Bruhweiler () half of the flux density at 6 cm is due to thermal electrons. They suggest that the rest of the flux density is due to 100-200 supernova remnants.

The eastern lobe has some characteristics of a hot spot at the terminal point of the jet expelled from a possible AGN in the centre of NGC 4569: is this evidence for a local cosmic ray acceleration and a possible connection (jet) to the disk. The available data exclude the existence of a present-day AGN in the center of NGC 4569 (see e.g. Barth & Shields or Gabel & Bruhweiler 2002). Instead, NGC 4569 harbors a low-ionization nuclear emission-line region (LINER). Thus its nucleus shows a very low activity. Furthermore, the western lobe does not resemble the classical AGN-expelled structure. It is a diffuse feature attached to the disk with vertical magnetic fields emerging from almost the whole inner disk. The possibility of an earlier AGN activity is discussed in the next section.

## A galactic wind in a ram pressure wind

In this section we discuss various possible origins of the observed radio-bubbles in NGC 4569. We start with geometrical aspects of the radio emission and an AGN scenario. We then analyse time scales for galactic outflows and galactic superwinds. For galactic superwinds we estimate energy needed to blow-up radio lobes and confront it with star formation in the disk and with two episodes of nuclear starburst. Finally we discuss the influence of cluster environments on the development of a ridge of radio continuum emission in the southern part of the western lobe.

### The radio geometry

A very surprising aspect of the radio lobes is their relative symmetry. NGC 4569 is moving rapidly within the hot intracluster medium. According to Vollmer et al. () the galaxy has passed the cluster core $\sim 300$ Myr ago and the current ram pressure is about $p_{\rm ram} \sim 2 \times 10^{-12}$ g cm$^{-1}$s$^{-2}$. Moreover, the galaxy is moving to the north-east. If the observed radio lobes were due to an AGN, one would expect the formation of a head-tail radio galaxy, because AGN jets are very vulnerable against forces perpendicular to the outflow direction, which is the case for a ram pressure wind. In contrast, in a galactic outflow/superwind a pressure driven bubble is created which expands in all directions and is thus much more stable against forces parallel to the galactic disk. We therefore suggest that the radio lobes are not due to an AGN (even now extinct), but due to a starburst-induced galactic outflow/superwind.

### Galactic outflows

We cannot yet determine reliably if the material in the flow leaves the potential of NGC 4569 (which is the defining property of a galactic wind) or if some or most of the material returns to the disk of NGC 4569 (which is typical for a galactic outflow). The radial velocity of the H$\alpha$ emitting gas cone at a radius of 4 kpc is measured to be about 100 km s$^{-1}$ (Bomans et al. ). The derived time needed to blow-up our radio lobe ($25-30$ kpc in size) is then about $\sim 200$ Myr, which is too long a life-time for the synchrotron electrons which do not show gradual spectral steepening. On the other hand it is possible that the magnetic field and cosmic rays are connected to a much hotter phase of the gas forming a superwind-like flow at considerably higher velocity.

### Superwinds

Typical outflow velocities of galactic superwinds are $\sim 700 -
1000$ km s$^{-1}$ (Heckman et al. ). So $\sim 30$ Myr are needed to develop the observed spatial extent of the radio lobes. In order to balance ram pressure, the inner pressure of the lobes has to be greater or equal to the external ram pressure $p_{\rm in} \ge p_{\rm
ram}$. Assuming the minimum energy condition we computed the total pressure of cosmic rays and magnetic fields in the lobes. A mean value inside regions delineated by the level of 1.2 mJy/b.a. at 4.85 GHz is $\approx 1-1.3\times 10^{-12}$dyn cm$^{-2}$. It rises to $\ge 2\times
10^{-12}$dyn cm$^{-2}$ when brighter parts of lobes inside the level of 4 mJy/b.a. are considered. Similar values are obtained when pressure balance conditions are assumed. These values are comparable in range to the ram pressure (see Sect. ). The joint magnetic and cosmic-ray energy density is also of order of 1.5 – $2\times 10^{12}$ erg cm$^{-3}$, again similar to the kinetic energy density of the wind. The outflows have thus a good chance to overcome the ambient gas pressure while the intergalactic wind can still deform the extended lobes (Sect. ).

To estimate the total energy requirements we assume a cone-like geometry for the outflow with a height of 25 kpc and a diameter of 15 kpc, the total volume occupied by the outflow is thus $V \sim
8\times 10^{67}$ cm$^{3}$. The minimum energy needed to overcome the external ram pressure within this region then is $E \sim p_{\rm
in} V \ge p_{\rm ram} V=10^{56}$ erg. With a typical supernova II total kinetic energy input $E_{\rm SN}=10^{51}$ erg we find that the total number of supernovae driving the outflow is $N_{\rm SN} \ge 10^{5}$. The mass ejected by the wind-type flow is $M_{\rm wind} = \rho_{\rm wind} V$, where $\rho_{\rm wind}$ is the mean gas density within the lobes. Assuming a mean outflow velocity of $v_{\rm wind}=700$ km s$^{-1}$ the mean density is $\rho_{\rm wind} \sim 2 E/{V v_{\rm wind}^{2}}=5.1 \times
10^{-28}$ g cm$^{-3}$ and the ejected mass is M$_{\rm wind} \sim 1.6 \times
10^{7}$ M$_{\odot}$. The fraction between the gravitational and the kinetic energy density is $\varepsilon_{\rm grav}/\varepsilon_{\rm in}
\sim \big(v_{\rm rot}/v_{\rm wind}\big)^{2}
\sim 0.1$, where we assumed a symmetric dark matter halo and a constant rotation velocity of $v_{\rm rot}=250$ km s$^{-1}$. Thus, a sufficiently strong superwind-type flow can easily overcome the gravitational potential of the galaxy and resist ram pressure.

### Disk star formation and the central starbursts

Is the star formation activity of NGC 4569 strong enough to explain such powerful wind-type flows? The disk of NGC 4569 is weakly forming stars. Due to the small star formation rate the arm-interarm contrast in the disk is low and the galaxy is classified as anemic by van den Bergh (). Thus the star forming H$\alpha$ disk (e.g. Koopmann et al. ) of NGC 4569, truncated by the recent ram pressure stripping event, cannot provide the necessary energy input to drive the observed radio lobes.

We are thus left with central starbursts. IUE ultraviolet and optical ground-based spectra of the nucleus of NGC 4254 of various dispersions were studied by Keel (). Narrow Balmer absorption lines and the overall shape of optical spectrum indicates that the optical light is dominated by a young starburst particularly rich in A-type supergiants. UV light is more peculiar and could not be modelled by the UV light from a dominant population of A supergiants. It was interpreted as an AGN or an additional unusually compact and extraordinary luminous central-star cluster. This last possibility was confirmed in the detailed study of HST/FOC UV data by Gabel & Bruhweiler (). Their spectral synthesis and photoionization analyses imply that this central starburst is very recent ($5-6$ Myr old) and consist of about $5
\times 10^{4}$ O and B stars packed in a region of $\approx 30$ pc in size. They also pointed out that the population of A supergiants discerned by Keel () must be older ($>15$ Myr) and by a factor of 10 more extended than the UV-bright core. This may mean that its content of OB stars (hence supernova progenitors) could be in the past considerably greater than $10^5$.

The central present day starburst is clearly too young ($5-6$ Myr) to be responsible for huge observed radio lobes (timescale of $\sim 30$ Myr). Also the number of OB stars generated by the recent starburst is less than the minimum number of supernovae II ($\sim
10^{5}$) needed to drive the observed radio structures. In contrast to that the timescale of the second, older starburst ($>15$ Myr), seen now as an abundant population of A-type supergiants matches the timescale of the observed outflow. This makes this old starburst (possessing at the past time also enough OB stars) better candidate to drive the observed radio lobes. However, a further detailed modelling of A supergiant-dominated population is needed to explain together with the OB dominated compact starburst the both UV and optical spectra and to constrain the starburst geometry, age and energetics, and to model the radio lobes formation. A detailed stellar population synthesis modeling of a past starburst which would match the present-day A-star content and produce enough OB stars at the time of its youth is required to state whether such an event is sufficient to energize the observed lobes.

### Radio ridge and ram pressure stripping

The horizontal ridge of radio continuum emission visible at 1.49 GHz in the southern part of the western lobe of NGC 4569 (Fig. ) coincides with a spot of polarized emission (Fig. ) parallel to this structure. The western side of the galactic disk is the side closest to the Virgo Cluster centre. In the ram pressure scenario of Vollmer et al. () NGC 4569 is moving within the Virgo intracluster medium to the north-east. Consequently, the ISM of NGC 4569 is pushed by ram pressure to the south-west (last frame of Fig. 6 of Vollmer et al. ). Thus, ram pressure is not directly responsible for the horizontal ridge of radio continuum emission.

We suspect the horizontal ridge of radio continuum emission to be outflowing gas of the galactic superwind colliding with the part of the stripped ISM of NGC 4569 which is still close to the galaxy. If this gas is atomic its surface density must be smaller than $10^{20}$ cm$^{-2}$, because it was not detected in the VLA data of Vollmer et al. (). On the other hand, if this putative south-western Hi plume is more extended than 20-30 kpc, its Hi mass cannot exceed several $10^{7}$ M$_{\odot}$, because it would have been detected with the Effelsberg 100-m telescope. The collision of outflowing and stripped gas leads to compression of the outflowing gas and the magnetic field contained in it, giving rise to the observed horizontal ridge of enhanced total power and polarized radio continuum emission. In addition, we expect that a shock is formed when the wind is hitting the relatively dense stripped ISM of NGC 4569. The observed flattening of the spectral index in the south-western part of the western radio lobe might then be due to in-situ particle acceleration in this large-scale shock.

The flat spectral index of the eastern lobe and the implied in-situ acceleration of relativistic electrons there might be due to a large-scale shock driven by the direct impact of ram pressure on this lobe, whereas the western lobe is protected from a direct impact of ram pressure by the stripped ISM of NGC 4569. This shielding might also be partly responsible for the different aspects of the two radio lobes. High resolution radio continuum observations are needed to confirm and resolve the distribution of the spectral index in the radio lobes. If our scenario is correct we would expect a flatter spectral index in the direction of the ram pressure wind, i.e. to the north-east in the eastern lobe.

Of course, our scenario has caveats: (i) why is the stripped and partly re-accreting gas seen in Fig. 6 of Vollmer et al. () located more to the west of the galaxy center than to the south-west? (ii) Can such gas, if it exists, efficiently shield/protect the galactic wind-like flows? Of what type is the stellar population, its age and extent responsible for radio lobes? More sensitive multi-frequency radio polarization and optical observations of NGC 4569 with high spatial resolution would greatly help to establish a detailed scenario of these peculiar gas outflows.

# Conclusions

Using the Effelsberg radio telescope working at 4.85 GHz and 8.35 GHz we discovered large scale outflows in the strongly stripped Virgo spiral galaxy NGC 4569. We found that:

-   NGC 4569 possesses large radio lobes extending up to 24 kpc from the disk, which is unusual for normal spirals and even more unusual for cluster spirals.

-   The lobes emit as much as 30% of the total flux density at 8.35 GHz.

-   The eastern lobe shows signatures of in-situ shock-driven cosmic ray acceleration.

-   The western lobe looks like a diffuse halo with vertical magnetic fields spread over the whole disk.

-   Using the continuum channels at 1.4 GHz from the Hi observations we suggest that its southern part shows effects of gas and magnetic field compression by the ambient intracluster medium.

The radio structure of NGC 4569 differs in many respects from that of normal spiral galaxies: the radio lobes with scales of tens of kiloparsecs are very rare phenomena in field spiral galaxies and have not yet been observed in a cluster spiral galaxy. The extraplanar western ridge of unpolarized and polarized radio continuum emission is also a peculiar feature, which is most probably due to the cluster environment.

The available data exclude present-day and past AGN activity in the core of NGC 4569. We exclude that the observed radio lobes are due to an extinct AGN, because NGC 4569 undergoes relatively strong ram pressure effects (Vollmer et al. ) which would lead to a head-tail galaxy. Instead, we propose that the observed radio lobes are due to a galactic superwind-like flows, induced by a starburst lasting several $10$ Myr. Such an event, which requires $10^5$ supernovae explosions and $10^{56}$ erg total input energy, is consistent with a stellar population dominated by a large number of A supergiants in the central region of the galaxy (Keel 1996). This is supported by estimates of the combined magnetic and cosmic-ray pressure inside the lobes from our radio data.

Our observations give the first evidence that galactic superwind-like flows can occur even in a spiral galaxy located near the cluster center (projected distance $=0.5$ Mpc.) They can expand to distances of several 10 kpc and might finally escape the galaxy's gravitational potential and enrich the intracluster medium with gas, dust and magnetic fields. In addition to magnetic fields expelled from galaxies during their interactions (Chyży & Beck ) objects like NGC 4569 may constitute an important source of intergalactic/intracluster magnetic fields, thereby relaxing the seed field problem in the dynamo theories (e.q. Widrow ).

Despite this first evidence, we are far from fully understanding the nature of the giant outflows in NGC 4569. High resolution, high sensitivity, multi-frequency radio continuum observations including polarization will greatly help to test our tentative scenario. The polarized radio emission detected from IC 3583 and the existence of a radio bridge connecting IC 3583 and NGC 4569 suggest a possible interaction between the two objects which will have to be addressed, too.

[^1]: We use a distance to the Virgo cluster of $D=17$ Mpc.

[^2]: The 100-m telescope at Effelsberg is operated by the Max-Planck-Institut für Radioastronomie (MPIfR) on behalf of the Max-Planck-Gesellschaft.


abstract: Convolutional Neural Networks (CNNs) have gained significant traction in the field of machine learning, particularly due to their high accuracy in visual recognition. Recent works have pushed the performance of GPU implementations of CNNs to significantly improve their classification and training times. With these improvements, many frameworks have become available for implementing CNNs on both CPUs and GPUs, with no support for FPGA implementations. In this work we present a modified version of the popular CNN framework Caffe, with FPGA support. This allows for classification using CNN models and specialized FPGA implementations with the flexibility of reprogramming the device when necessary, seamless memory transactions between host and device, simple-to-use test benches, and the ability to create pipelined layer implementations. To validate the framework, we use the Xilinx SDAccel environment to implement an FPGA-based Winograd convolution engine and show that the FPGA layer can be used alongside other layers running on a host processor to run several popular CNNs (AlexNet, GoogleNet, VGG A, Overfeat). The results show that our framework achieves 50 GFLOPS across 3 $\times$ 3 convolutions in the benchmarks. This is achieved within a practical framework, which will aid in future development of FPGA-based CNNs.
author:

bibliography: fpt.bib
title: Caffeinated FPGAs: FPGA Framework For Convolutional Neural Networks

# Introduction

Convolutional Neural Networks (CNNs) are highly accurate deep learning networks inspired by the mammalian visual cortex. A number of works explored the implementation of CNNs on FPGAs  to take advantage of their low-power, customizable and programmable fabric. While FPGA implementations show promise in efficiently computing CNNs, they lack the infrastructure available for both CPUs and GPUs. This makes FPGAs inaccessible to deep learning scientists. There are many frameworks for CNN implementations, most of which provide support for CPU, GPU or the option of both . These frameworks allow the programmer to launch any CNN model, and contain comprehensive tests for both layer-based and system-based executions . However, none of the prominent CNN frameworks provide support for FPGA implementations. As a result, to implement a CNN on the FPGA, the designer has to manually design the implementation for each model, as well as test for correctness and optimize for performance, essentially rebuilding from scratch, rather than taking advantage of existing work.

CNNs are very computationally intensive with most of the computation in the convolution layers. The convolution layers require a large number of multiply-accumulate operations. The large computational complexity motivates efforts to reduce the number of required operations. One approach is to use the FFT for convolution because in the frequency domain, the convolution becomes multiplication of each transformed input with the corresponding transformed filter coefficients, resulting in a compute reduction and speedup  . An alternative approach uses the Winograd minimal filtering algorithm to take advantage of the overlapping computations between adjacent convolution windows . In this work we implement and optimize the Winograd algorithm on an FPGA within the Caffe framework .

This paper makes the following contributions:

-   We present an adaptation of the Caffe CNN framework with support for the Xilinx FPGA SDAccel environment. This adaptation allows us to launch CNN classification on CPU-FPGA-based systems.

-   We describe a modification to the Winograd convolution algorithm to further reduce DSP utilization for FPGA-based implementations.

-   We implement the Winograd convolution algorithm targeting any $3\times3$ convolution layer with unity stride and benchmark it across several popular CNNs. Results show that the architecture achieves approximately 50 GFLOPS across the $3\times3$ convolution layers of the benchmark suite, while using 83.2% of the available SDAccel resources in a Xilinx Virtex 7 XC7VX690T-2.

-   Finally, the software and hardware implementation details have been made open-source and can be found at the following link: <https://github.com/dicecco1/fpga_caffe>.

This work is organized as follows: Section  provides background information on CNNs and the Xilinx SDAccel OpenCL framework. Section  discusses the Winograd convolution algorithm and FPGA implementation. Section  details the features included in the FPGA Caffe framework and Section  shows the area utilization and performance results of the Winograd convolution engine within the FPGA Caffe framework. Section  reviews related work and compares this work to other recent FPGA implementations. Finally, Section  discusses future work related to the FPGA Caffe framework, and Section  concludes the paper.

# Background

The following subsections detail the necessary background information regarding CNNs and the Xilinx SDAccel OpenCL development environment.

## Convolutional Neural Networks

CNNs are a popular type of supervised machine learning algorithm. Similar to other machine learning algorithms, CNNs can be trained using back propagation to learn complex representations useful for many applications. CNNs are commonly used for performing object recognition in pixel-based input. A popular CNN model such as AlexNet  can be used to classify up to 1000 different objects in images with high accuracy.

These networks have two different modes of operation: training and inference. In the case of object recognition, training involves feeding a large number of human-annotated images into the network. These images are used by the CNN to repeatedly update the model's weights and biases such that it can learn how to recognize the objects in the human-annotated images. Classification uses the trained CNN model and presents it images that the model has never seen to attempt to predict what the object in the image is.

## CNN Layers

AlexNet was one of the first successful ImageNet submissions employing CNNs and it consists of several layers: Convolution, Max Pooling, Fully Connected (FC), Rectified Linear Unit (ReLU), and Local Response Normalization (LRN) . In many of the top performing CNNs, Convolution, Pooling, Fully Connected, and ReLU layers are typically used, while LRN and other layers are sometimes used depending on the model . The general structure of a CNN usually consists of stacks of convolution layers with ReLU activations followed by a pooling layer. In recognition applications, fully connected layers are used towards the output to reduce spatially organized information into a decision. Convolution layers represent the majority of the computation for a CNN, with extreme cases requiring upwards of 30 GFLOPs of computation . The ReLU layer is an activation function used to introduce a non-linearity into the network and can be described by $y=\max(0, x)$, which is applied to every data point of the input. Pooling is a simple reduction operation, such as max or average, applied to local regions of the input using a sliding window approach. FC layers consist of dense connections between neurons and usually contain the majority of the weights in the network. The computation of FC layers corresponds to a matrix multiplication followed by the addition of an optional bias parameter to each output.

## Parallelism Strategies

Given the large computational requirements and inherent parallelism of neural networks, these architectures are ideal for hardware accelerators. Popular parallelism strategies can be reduced to three main categories.

– splitting the data across different execution threads, but using the same model. For the pixel-based input (e.g. images) natural to CNNs, data parallelism is inherent given the independence of individual and local groups of pixels. Fine-grained data parallelism can be applied using operations applied concurrently to all pixels, while coarse-grained data parallelism can be applied by processing "minibatches" of hundreds or thousands of input images during training.

– splitting the model across different execution threads, but using the same data. This strategy offers several advantages, such as being able to accommodate large neural network sizes by splitting the weights across hardware accelerators, and employing a type of efficient model averaging during training.

– operating different dependent steps of computation concurrently on different threads, so that output from one step is streamed as input to the next, while execution of steps is overlapping. The feed-forward computation of CNNs are well suited for pipeline parallelism, so hardware that can exploit deep pipeline parallelism (e.g. FPGAs) can offer an advantage.

## SDAccel OpenCL FPGA Programming Model

Using the SDAccel OpenCL environment to perform computations on an FPGA involves both host and kernel code. The host code is used for programming the FPGA, passing data between the host's memory and the FPGA's global memory, and launching the kernel on the FPGA. The FPGA is segmented into two regions, the programmable region and the static region. The static region is programmed upon power-up and it contains the interfaces to global memory and PCIe. The programmable region contains the kernel, the computation to be accelerated. The kernel code is synthesized into hardware and configured into the programmable region of the FPGA. The synthesized kernel can contain one or more compute units (CUs), where a CU corresponds to the hardware unit responsible for the required computation. One approach to increasing parallelism in the host code is to instantiate multiple CUs as shown in Fig. , with each CU handling an equally sized portion of the problem . The portion of the problem handled by a single CU is referred to as the local work group size, while the size of the overall task to be completed is referred to as the global work group size. Each CU has its own local memory that is only accessible to the CU while all CUs share the global off-chip memory of the platform.

We have integrated SDaccel into the Caffe framework and used it to develop a convolution kernel using the Winograd transform.

# Winograd Convolution

Winograd convolution exploits the Winograd minimal filtering algorithm . This approach has been shown to reduce the amount of required floating point operations . The sections below provide an overview for the Winograd convolution algorithm and discusses the implementation details of the FPGA-based Winograd convolution engine.

## Winograd Convolution Algorithm

The Winograd convolution algorithm output is referred to as $F(m \times m, r \times r)$. In this expression $m \times m$ refers to the output tile size for a given input tile, meaning that $m \times m$ output values are produced for every instance of $F(m \times m, r \times r)$. The filter size in this case is $r \times r$. For a given filter size, many different values of $m$ can be chosen, which changes the computational complexity of $F(m \times m, r \times r)$, but does not impact the overall result. In this work, we implement a $F(2 \times 2, 3 \times 3)$ Winograd algorithm. In the case of $3\times3$ convolutions, $F(2 \times 2, 3 \times 3)$ has been shown to provide significant performance gains for GPU implementations in , though larger tile sizes may produce additional gains. This work targets $F(2 \times 2, 3 \times 3)$, mainly due to its simplicity of implementation however future work may explore other tile sizes as well. The equations for Winograd convolution are shown in Equations  to . Equation  shows how the filter transformation is calculated.

$$\label{eq:winograd_u}
U = GgG^{T}$$

Where

|     |                                                           |
|----:|:----------------------------------------------------------|
| $U$ | is a $(m + r - 1) \times (m + r - 1)$ transformed filter; |
| $g$ | is an $r\times r$ filter;                                 |
| $G$ | is an $(m + r - 1) \times r$ transform matrix, defined by |
|     | the Winograd algorithm.                                   |
|     |                                                           |

The filter values ($g$) are known at compile time and remain constant during run-time. Therefore, to save resources during run-time, the Winograd transformation for the filter values, shown in Equation , can be executed at compile time, on the CPU. This approach saves FPGA resources. However, pre-computing filter values increases the memory storage requirement. For direct convolution, the $3\times3$ filters require $C\times 3 \times 3$ storage, where $C$ is the number of input channels. For Winograd-based convolution, after the transformation, the $3\times3$ filter is transformed into a $4\times4$ matrix, requiring $C\times4\times4$. Therefore the storage requirement is increased by 33%.

Equation  shows how the input transformation is calculated.

$$\label{eq:winograd_v}
V = B^{T}dB$$

Where

|     |                                                          |
|----:|:---------------------------------------------------------|
| $V$ | is an $(m + r - 1) \times (m + r - 1)$ transformed data  |
|     | tile;                                                    |
| $d$ | is an $(m + r - 1) \times (m + r - 1)$ input tile;       |
| $B$ | is an $(m + r - 1) \times (m + r - 1)$ transform matrix, |
|     | defined by the Winograd algorithm.                       |
|     |                                                          |

For $F(2 \times 2, 3 \times 3)$ input tile ($d$) is $4\times4$ and is generated by a sliding window across the 2-D input feature data. Shown in Fig. , the $d$ window slides horizontally across the input data, with a stride of two. After the Winograd transformation, the $4\times4$ $V$ tiles are stored back into memory. Because the input tiles contain overlaps of the input data, four times more memory storage is required.

Equation  shows how the pre-computed transformed filter $U$ and the run-time transformed input data $V$ are used to calculate the final output, a $2\times2$ tile $Y$. Each Y tile corresponds to a $2 \times 2$ non-overlapping subsection of the overall convolution output.

$$\label{eq:winograd}
Y = F(m \times m, r \times r) = A^{T}[U \odot V]A$$

Where

|         |                                                           |
|--------:|:----------------------------------------------------------|
|     $Y$ | is an $m \times m$ output tile;                           |
| $\odot$ | is an element wise multiplication;                        |
|     $A$ | is an $(m + r - 1) \times m$ transform matrix, defined by |
|         | the Winograd algorithm.                                   |
|         |                                                           |

## FPGA Winograd Convolution

To validate the FPGA Caffe framework, efforts were primarily focused on implementing the Winograd algorithm for convolution ($F(2 \times 2, 3 \times 3))$ discussed in Section , though other less optimized implementations of the layers discussed in Section  have been created as well. The architecture created for the Winograd algorithm on the FPGA can be separated into three stages of operation: input, compute, and output.

**Input Stage** – used to move the input frames from off-chip SDRAM memory to the on-chip BRAMs. A portion of the input frame is burst read into a temporary buffer through an AXI request and moved into tiles of four rows by two columns stored in BRAM. The tiles in this case deviate from the algorithm discussed in Section  as data is only replicated in overlapping rows of tiles rather than both overlapping rows and columns to reduce the memory overhead of storing input tiles. This tiling method is similar to the method in Fig. , however while the stride stays the same, the tile width is two resulting in no overlaps. To facilitate the compute stage and output stages, the number of tiles per row is set to be a multiple of eight, with padding added as required. After tiling is completed, a portion of the Winograd input transformation shown in Equation  is applied to every column of the $4\times2$ tiles.

$$\label{eq:wino_col}
\begin{split}
O[0,j] &= I[0,j] - I[2,j]\\
O[1,j] &= I[1,j] + I[2,j]\\
O[2,j] &= I[2,j] - I[1,j]\\
O[3,j] &= I[1,j] - I[3,j]\\
\end{split}$$

Where

|           |                                                     |
|----------:|:----------------------------------------------------|
| $I[n, m]$ | is the input of the partial transform at $(n, m)$;  |
| $O[n, m]$ | is the output of the partial transform at $(n, m)$. |
|           |                                                     |

This partial transformation is replicated eight times in the input stage such that eight columns of the input tiles can be processed per cycle to reduce the overhead that this preprocessing causes.

The full input transformation in Equation  requires a total of eight instances of Equation  per $4 \times 4$ tile, with four instances being applied to the columns, and four instances being applied to the rows of the tile (indices of Equation  are swapped for row calculations) as shown in Fig. . This results in 32 floating point additions and 64 DSPs per input tile transformation instantiation. Exploiting the fact that each input tile overlaps every two rows and every two columns with its neighbor, savings can be achieved by first precomputing either all of the column wise instances of Equation  or all of the row wise instances rather than computing the full transform for each tile. This reduces the number of partial transforms required for either the column wise or row wise instances from four to two per tile, which reduces the number of floating point additions to 24 from 32 per input tile transformation (with potentially one additional set of column transformations for the edge of each row).

**Compute Stage** – where the bulk of the processing time is spent in the architecture. Following the input stage, each 4x2 tile and its neighbor are fed into a pipelined processing element that handles all subsequent computations required by the algorithm. The processing element completes the remaining set of partial transforms required after the input stage, performs the element wise multiplication between the input and the weights, computes the output transform, and accumulates the result within an output buffer. This process is repeated with different weights per output feature map until all of the output feature maps have been computed. Per output feature map this requires $C\times P\times Q + D$ clock cycles, where C is the number of input channels, P is the number of output tiles per column, Q is the number of output tiles per row, and D is the number of cycles required to fill the pipeline. To reduce the cycles required per output feature map, the processing element is replicated four times, allowing this stage to effectively be completed four times faster (assuming that D is small).

**Output Stage** – handles transferring the output frame back into the off-chip DRR memory. First the results are gathered from the partial result buffers of the compute stage to an output buffer. Then the output is burst written to the DDR through AXI.

To further improve the performance of the engine, it has been replicated such that there are two CUs rather than one. Each CU handles a separate input image to exploit coarse-grained data parallelism. The performance of the engine is directly proportional to how many CUs can be replicated, with the execution time being dictated by the equation $T = (C \times N)/ (F \times \#CUs)$, where F is the operating frequency, C is the number of clock cycles, N is the number of images, and T is the total latency.

# FPGA Caffe Framework

The Caffe framework  is used to describe CNNs based on predefined layer implementations with CPU and GPU backends. In this section we describe our approach to augmenting the Caffe framework to enable CNN classification using FPGAs. The discussion below will detail how memory transfers between the device and host are handled, how FPGA test benches may be used within the framework, and several FPGA specific layer implementations. The layers include a custom layer for reprogramming the FPGA and pipeline layers for fused layer implementations.

## Caffe Model Description

The infrastructure in Caffe allows for simple description of common layers used in CNNs and provides several implementations of existing high performance CNNs as well. Each layer in Caffe corresponds to a set of computations required by a given CNN model, allowing for modular CNN implementations. Caffe also allows for networks to be defined without modification of source code by providing model definitions through the Protocol Buffer Language . This allows for networks to be constructed through a file describing which layers are required and their respective ordering.

The model description format in Caffe has been augmented in this work to support additional features described in the sections below. Namely, in the FPGA Caffe framework a program layer can be specified in any position of the network to force the FPGA to be reprogrammed, pipelined layers can be specified if a fused layer is required, and the Winograd convolution engine discussed in Section  can be specified when needed.

## OpenCL Brew

In Caffe a Brew is referred to as a mode of operation that determines the target architecture on which CNN classification or training is executed. The original Brews are CPU or GPU, with the CPU Brew containing the C++ infrastructure required to define layers using a CPU, and the GPU Brew providing similar features but for NVIDIA GPUs using CUDA and cuDNN . For each Brew, Caffe contains test cases available for every layer, allowing for fast determination of functional correctness and benchmarking.

This work extends the baseline Caffe framework to include the OCL (OpenCL) Brew, which provides support for Xilinx FPGA-based CNNs and could easily be adapted to target Altera's OpenCL programming environment as well. The user can choose between the different Brews by building the framework using the corresponding Makefile flags and changing the Brew to OCL. Fig.  shows an overview of the augmented system with the OCL Brew, where inputs and outputs are the same as in the CPU and GPU Brews, but the underlying hardware of the system is comprised of the CPU for host code and the FPGA for layer computations.

To perform a forward pass (inference) using the OCL Brew, we added an API call: forward_ocl(). The forward_ocl() API call is used as the forward operator on the condition that the Brew is OCL and the function is defined, otherwise it defaults to the forward_cpu() call as in the baseline Caffe implementation.

## OpenCL Memory Management and Synchronization

Data in Caffe is represented as a four-dimensional flattened array, with allocation, resizing, and synchronization between CPU and GPU resources abstracted from its usage . The memory management API in Caffe handles synchronization between the host and GPU devices such that memory is only transferred back to the host when necessary. To accomplish this, the state of the memory is stored as either HEAD_AT_GPU, HEAD_AT_CPU, or SYNCED which is verified upon accessing the data. If the state of the data is HEAD_AT_GPU and the host requests the data, a data transfer from the device to the host will be issued and the state will change to SYNCED.

Support for memory synchronization between the host and the FPGA in the FPGA Caffe framework builds on the memory synchronization features described above. To accomplish similar functionality, OpenCL APIs are used with an additional object corresponding to the FPGA device memory object for each data structure. When data is passed from the host to the FPGA, the state of the memory changes to HEAD_AT_OCL such that on subsequent accesses it will either stay in the device memory or be transferred back to host memory. If the data is required by the host, a memory transfer will be issued from the device to the host and the state of the memory will change to SYNCED. To access the FPGA memory object, calls to either mutable_ocl_data() for modifying data (layer output data) or ocl_data() for static data (layer input data, such as weights), are required. These two functions were added to Caffe to handle both the creation and synchronization of the device and host memory while maintaining transparency of memory manipulation as in the baseline Caffe implementation.

## FPGA Testbenches

Testing a given layer in the FPGA framework can be accomplished in two different ways depending on the stage of development. Layers can be tested using individual test cases through the test framework provided in Caffe. Alternatively they can also be tested through the use of standalone host code by invoking only the host code required to launch the kernel. In either case, the layer can be tested using a hardware implementation or using software emulation based implementations created in the Xilinx SDAccel environment.

The baseline Caffe framework has a number of tests that are available for each layer of the system . Each test can be made into a test for the FPGA implementations by changing the Brew to OCL and modifying parameters to suit a given layer. These tests allow for larger scale testing to verify that the layer has been integrated properly within the Caffe framework. Aside from providing breadth to the test suite, this also allows for fast prototyping of layers through the use of software emulated layers provided by the capabilities of SDAccel .

## Kernel Compilation

Compiling kernels for CPUs and GPUs amounts to compiling programs into instructions that program the hardware, whereas compiling for FPGAs involves synthesizing full circuits. As a result, the overhead of compilation for FPGAs (hours) is much greater than that of CPUs and GPUs (milliseconds), and so runtime compilation of FPGA kernels is not possible. We deal with this problem by employing an offline compilation strategy, where deep learning practitioners can make use of precompiled binaries at run-time.

## XCLProgram Layer

Though FPGA Caffe makes use of offline binary compilation, there is still significant overhead from programming the FPGA (100-300 ms) compared to the CPU and GPU (0.001-0.005 ms). This programming overhead conflates the measured execution time for a layer in the Caffe benchmarking functionality. We introduce a new layer, the XCLProgram layer, as a method for giving the user greater control over how the FPGA is programmed, as well as the ability to separately benchmark the execution time of each layer and programming overhead. The XCLProgram layer as input receives a pointer to the FPGA binary file, as well as the kernel name.

## Pipelined Layers

In the GPU-based approach native to Caffe, modularity is enforced layer-wise, meaning before each layer is executed the GPU is programmed with the appropriate kernel and memory is synchronized with the host. In FPGA Caffe this becomes a bottleneck given the large overhead of programming the FPGA, and requiring such frequent memory synchronization with the host is much more expensive on FPGAs compared to high memory bandwidth GPUs. Additionally, this modularity used in Caffe limits parallelism strategies to within each kernel. To address these issues, we introduce a new layer type in FPGA Caffe called pipeline layers. Facilitated by XCLProgram layers, pipeline layers package multiple kernels into a single binary, with kernel-kernel communication occurring through local memory structures on the FPGA (i.e. FIFO). Pipeline layers reduce the number of times the FPGA is programmed, and eliminate the need to synchronize memory with the host between every layer. Most importantly, pipeline layers allow pipeline parallelism strategies across layers, increasing throughput by allowing multiple layers to execute concurrently. While the use of pipeline layers violates some modularity assumptions of Caffe, we argue that this is practical given that combinations of layer groups are very predictable in practice (e.g. convolution, ReLU, pooling).

# Results

This section describes our study of the Winograd Convolution Algorithm and the results gathered from using FPGA Caffe to implement the $3\times3$ convolution layer using the Winograd Convolution Algorithm. The platform we use includes an Alpha-Data ADM-PCIE-7V3 card with a Xilinx Virtex 7 XC7VX690T-2 running at 200 MHz and an Intel Xeon CPU E5-2620 running at 2.0 GHz for the host application code. The Xilinx Virtex 7 is contained within a server that has been virtualized to support virtual machines (VM) and is connected through PCIe. The VM in use has 8GB RAM and four cores. An Intel i7-4770k running at 3.5 GHz was used for CPU comparison and an nVidia Quadro K620 for GPU comparisons. The Xilinx SDAccel version number is 2015.1.3, CUDA version number is 7.5, cuDNN version number is 4.0  and the CPU host code uses OpenBLAS  with eight threads enabled. The CPU, GPU, and FPGA implementations all use single precision floating point as their data representation.

## Winograd Resource Utilization

The resource utilization post place and route is shown in Table . The highest utilization post place and route for both CUs is the LUT utilization at 83.2% of the SDAccel region's available LUTs. The utilization post place and route accounts for additional resources required to integrate into the SDAccel framework, which drives the significant LUT utilization in comparison to other resources on the device.

|                          |         |         |       |       |     |     |
|:------------------------:|:-------:|:-------:|:-----:|:-----:|:---:|:----|
|            \-            |   FF    |   LUT   |  DSP  |       |     |     |
|          (18Kb)          |         |         |       |       |     |     |
|                          |         |         |       |       |     |     |
|   Total Resources (A)    | 866,400 | 433,200 | 3,600 | 2,940 |     |     |
|                          |         |         |       |       |     |     |
|        Region (B)        | 551,040 | 275,520 | 2,376 | 1,940 |     |     |
|                          |         |         |       |       |     |     |
|     Conv. Engine (C)     | 253,873 | 229,226 | 1,307 | 1,188 |     |     |
|                          |         |         |       |       |     |     |
|    Utilization (C/B)     |  46.7%  |  83.2%  |  55%  | 61.2% |     |     |
|                          |         |         |       |       |     |     |
| Device Utilization (C/A) |  29.3%  |  52.9%  | 36.3% | 40.4% |     |     |

Single Precision Winograd Convolution Engine Resource Utilization Post Place and Route

Given the utilization of the device resources, further instances of the CU could theoretically be added, though the limits on the available resources in the SDAccel region of the FPGA shown in Table  makes it impossible to place more than two CUs. This in turn limits the potential performance of the architecture within the SDAccel environment because of the overhead of the static region and the lack of resources available in the reconfigurable region.

To quantify the DSP savings from using the input stage column transformation discussed in Section  we consider three separate cases. The first case is that the full input transform is pre-computed for each four by four tile before sending the data to the processing elements of the compute stage, with only one instance of the full input transformation in Equation . The second case is that the full input transformation in Equation  is computed within each processing element of the compute stage using one full input transformation per processing element. Finally, the last case is the one discussed in Section , in which a column-wise partial transformation is computed for all input tile columns, using eight partial transformations to reduce computation overhead, with the remaining partial transformations computed within the processing elements using four partial transformations per processing element. Table  shows the resource utilization of the three cases post place and route. Between cases 2 and 3 there is a DSP savings of 61 units and a decrease in LUT usage. This is slightly less than what is anticipated in Section , though the difference can be attributed to fewer addressing calculations being required in case 2 due to the elimination of the partial transformations in the input stage. When comparing with case 1, the DSP utilization is approximately 20% less than case 3, however the BRAM utilization has increased by 33% due to the storing of replicated data in overlapping tiles. While the LUT usage and DSP utilization is better in case 1, the BRAM utilization makes it difficult to place and route more than 1 CU, as the BRAM utilization for two would require most of the available BRAM.

|                 |         |         |     |     |     |
|:---------------:|:-------:|:-------:|:---:|:---:|:---:|
|      Layer      |   FF    |   LUT   | DSP |     |     |
|     (18Kb)      |         |         |     |     |     |
|                 |         |         |     |     |     |
|  Pre-Transform  | 143,965 | 127,989 | 523 | 914 |     |
|                 |         |         |     |     |     |
| Transform in PE | 166,905 | 153,887 | 715 | 688 |     |
|                 |         |         |     |     |     |
| Transform in PE | 158,266 | 145,892 | 654 | 688 |     |

Resource Utilization for Different Winograd Strategies With One CU

## FPGA Caffe Benchmark Results

To evaluate the Winograd convolution engine within the FPGA Caffe framework, a set of benchmark CNNs is required to view its performance across varying workloads (number of output feature maps and output sizes). The benchmark suite that we use is adopted from the Soumith Chintala convnet-benchmarks , which is composed of previous ImageNet winners including: AlexNet , VGG A , Overfeat , and GoogleNet . Due to the RAM size of the virtual machine used for the host code, the batch size for each benchmark is reduced by half to fit within the host VM.

Table  shows the performance of the system in comparison to both CPU and GPU implementations of the $3\times3$ convolution layers of each CNN. To calculate the GFLOPS of the Winograd convolution engine, the number of floating-point operations is taken to be the same as direct convolution, as is the case in , which is considered to be the effective GFLOPS. Comparing the geometric averages in Table , the Winograd convolution engine performs approximately 2.1 times slower than the CPU implementation and 9.4 times slower than the GPU implementation.

```latex
\begin{table*}[ht]\centering
\caption{CPU, GPU, and FPGA 3$\times$3 Convolution Benchmark Results}
\begin{threeparttable}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|l|} \hline
Network & \shortstack{CPU Run \\ Time (ms)} & \shortstack{GPU Run\\ Time (ms)} & \shortstack{FPGA Run \\Time (ms)} & \shortstack{CPU GFLOPS} & \shortstack{GPU GFLOPS} & \shortstack{FPGA GFLOPS}\\ \hline
AlexNet (64 Images) & 492 & 261.2 & 1,010 & 94 & 177.1 & 45.8 \\ \hline
VGG A (32 Images) & 4,310 & 745.14 \tnote{a} & 8,713 & 111.2 & 642.9 & 55\\ \hline
Overfeat (64 Images) & 2,030 & 387.1 & 4,781 & 139.2 & 730.2 & 59.1\\ \hline
GoogleNet (64 Images) & 1,506 & 209.66 \tnote{b} & 2,937 & 81.8 & 587.8  & 42\\ \hline
Geometric Average & 1,595.6 & 354.51 & 3,333.9 & 104.46 & 470.17 & 50\\ \hline \end{tabular}
\begin{tablenotes}
\item[a] GPU memory could only fit 8 images, so the model was run with 8 images and execution time was multiplied by 4 to get a performance estimate.
\item[b] GPU memory could only fit 32 images, so the model was run with 32 images and execution time was multiplied by 2 to get a performance estimate.
\end{tablenotes}
\end{threeparttable}
\label{table:timing}
\end{table*}
```

# Related Work

There exist several frameworks for implementing CNNs depending on the targeted platform and programming language, with many of the popular frameworks discussed in . These frameworks typically support a number of programming languages including: C/C++, Matlab, Python, etc. With respect to Caffe, there is an independent effort underway to add OpenCL support , though this support is meant primarily for exposing more GPUs to Caffe rather than FPGAs. The work in  provides additional functionality to support AMD devices through OpenCL and provides some similar features to those explored in this work. However, these works are differentiated in that the OpenCL implementations are abstracted away from being GPUs in this work through a separate Brew for OpenCL to allow for implementations across many different compute engines. Additionally, given the non-standard development model currently supported in FPGA OpenCL tools, frameworks that support standard OpenCL will not work with FPGAs without significant framework modification. This is the case primarily because the FPGA OpenCL tools require offline compilation of kernels and vendor-specific attributes to achieve suitable performance. The framework introduced in this paper allows for the support of FPGAs through layer-specific implementations. These include the ability to specify when to program the FPGA, the addition of pipelined layers, and precompiled FPGA-specific layer implementations.

Table  shows the performance of this work compared to several recent FPGA works. The highest performing implementations are Qiu  and Suda , which is enabled by their use of fixed point representations. The work of Zhang is approximately 1.2-fold higher performance than this work while using the same data representation, though as expected our DSP utilization is significantly lower while achieving comparable throughput. While the performance of this implementation is lower, it does not require precision analysis prior to usage and it does not need to be resynthesized for new work loads as is the case in all of the prior works.

```latex
\begin{table*}[ht]\centering
\caption{Comparison Between Existing FPGA Works}
\begin{tabular}{|c|c|c|c|c|l|} \hline
Metric & Zhang~\cite{zhang} & Suda~\cite{suda} & Qiu~\cite{qiu} & This Work \\ \hline
Clock Frequency (MHz) & 100 & 120 & 150 & 200 \\ \hline
Precision & 32 bit float & 8-16-bit fixed & 16-bit fixed & 32 bit float \\ \hline
FPGA Version & Virtex 7 VX485T & Stratix-V GSD8  & Zynq XC7Z045 & Virtex 7 XC7VX690T-2 \\ \hline
DSP Utilization & 2,240 & (Not specified) & 780 & 1,307 \\ \hline
Host Connection & Microblaze, on chip host & PCIe & ARM Cortex-A9 Processor, on chip host  & PCIe \\ \hline
GFLOPS/GOPS & 61.62 & 136.5 & 187.8 & 50 \\ \hline
\end{tabular}
\label{table:fpga_compare}
\end{table*}
```

# Future Work

Future work related to this framework will contribute to both performance and usability. First, completing the implementation of back propagation will ensure FPGAs can be used for both classification and training, where hardware acceleration is especially important. Given the modularity between the solver, network, and layer in Caffe, only the layers need to be modified to accommodate backward propagation. The structure of the network that defines the collection of layers, as well as the operation of the solver that calls the backward methods to generate gradients and perform a weight update, is already in place.

As well, experimenting with reduced-precision implementations of common layers is an important next step. Recently, GPUs have started to support half precision based implementations of CNNs  and many of the FPGA works have been focused on reduced precision implementations . Half precision or fixed-point implementations both would offer significant area gains for FPGA implementations as shown in , as a floating-point multiplication requires three DSP units and additional LUTs and flip-flops for current Xilinx FPGAs, while half precision or fixed-point multiplications require only one DSP unit and significantly fewer LUTs and flip-flops. This would allow for the replication of both processing elements and CUs to improve performance.

Finally, multi-FPGA-based parallelism strategies are crucially important in scaling up to accommodate larger data and model sizes. Current solutions involve using GPU clusters with Infiniband interconnects and MPI, which allow fast node to node data transfer and increase parallelism capabilities . The use of FPGAs are attractive in this domain given the flexibility and high performance/watt, and FPGAs can benefit from much of the work being done investigating multi-GPU parallelism strategies.

# Conclusion

In this work we presented a framework for implementing CNNs using FPGAs based on the Caffe CNN framework. The framework allows for transparent support for individual FPGA implementations of layers for testing and verification. This framework was validated by implementing the Winograd convolution layer and testing it across several CNNs. The results show that with 83.2% of the available SDAccel resources we are able to achieve 50 GFLOPs across the $3 \times 3$ convolution layers of four different CNNs. While this does not improve upon current implementations in terms of performance, it demonstrates the capabilities of the framework, which allows for further work that could lead to higher performance.

# Acknowledgment

The authors would like to thank Xilinx, CMC and emSYSCAN, NSERC, and CFI for the funding and resources provided for this work.