author: R. Tylenda, T. Kamiński, and M. Schmidt
date: Received; accepted
title: An analysis of a spectrum of V838 Monocerotis in October 2005

We attempt to obtain information about the structure and evolution of the object in the decline from the 2002 eruption. The results of spectroscopic observations of the object obtained in October 2005 with the Keck/HIRES instrument, presented in detail in Paper I, are analysed and discussed. Modelling of the observed line profiles has been used to constrain the physical parameters of the system.

The kinematics of the atmosphere of V838 Mon is very complex. Our analysis of the molecular bands and the P-Cyg profiles of atomic lines shows that the object loses matter with a velocity of up to 215 km s$^{-1}$ and a rate of $10^{-6}-10^{-5}$ M$_{\sun}$ yr$^{-1}$. In the profiles of some atomic lines, we have, however, found evidence of matter infall. Moreover, a narrow absorption component, which is particularly strong in some P-Cyg profiles, may indicate that a jet-like outflow has also been formed.

We show that the observed emission in the \[\] lines and an eclipse-like event observed in November/December 2006 was probably caused by interactions of the expanding matter, ejected by V838 Mon in 2002, with radiation from the B3V companion. In particular, the observed profiles of the \[\] lines can be easily modelled in this scenario and allow us to estimate parameters of the system, such as the position of the B3V companion relative to V838 Mon and the line of sight, density in the outflowing matter, and mass lost in the 2002 eruption. The observed appearance of strong H$\alpha$ emission, just before and during the eclipse-like event, can be interpreted as a result of the accretion of the outflowing matter onto the B3V companion: the accreted matter, shocked above the stellar surface, can be a source of extreme-UV and soft X-ray radiation capable of ionizing and exciting H in the outflow.

# Introduction

The eruption of V838 Monocerotis was discovered at the beginning of January 2002. As observed in the optical, the eruption lasted about three months . During the event, the object reached a luminosity of $\sim 10^6~L_\odot$ . After developing an A–F supergiant spectrum at the maximum in the beginning of February 2002, the object evolved to lower effective temperatures and in April 2002 was practically unable to be detected in the optical, remaining very bright however in the infrared. Optical spectroscopy acquired at that time discovered a B3V companion of the erupted object . analysed the evolution of V838 Mon during outburst and early decline.

Different outburst mechanisms, including an unusual nova, a He-shell flash, and a stellar merger, were proposed to explain the eruption of V838 Mon. These mechanisms were critically discussed by , the authors conclude that the only mechanism that could satisfactorily account for the observational data was a collision and merger of a low-mass pre-main-sequence star with an $\sim 8\,M_\odot$ main-sequence star.

In (hereinafter referred to as Paper I), a high resolution spectrum of V838 Mon acquired with the Keck I telescope in October 2005 was presented. In the present paper, we analyse and discuss the results obtained from this spectrum.

# Evolution of V838 Mon during the decline after the 2002 eruption

A few months after the eruption, discovered at the beginning of January 2002, V838 Mon entered a relatively calm decline phase. V838 Mon then resembled a very cool oxygen-rich (C/O $<1$) supergiant slowly declining in luminosity . It dominated the observed spectrum at green, red and infrared wavelengths. The blue part of the spectrum was dominated by the light of the B3V companion.

Late in 2004, narrow emission lines, belonging mostly to \[\], appeared mainly in the blue part of the spectrum , and strengthened in time . During November–December 2006, an eclipse-like event of the B3V companion was observed (). At the epoch of the event, the \[\] emission lines reached their maximum strength and strong emission in Balmer lines appeared .

In October 2005, we obtained an optical spectrum of V838 Mon using the Keck/HIRES instrument. Results of these observations were presented in Paper I. The star V838 Mon itself is seen as a very cool supergiant that dominates the green and red parts of the spectrum. Numerous, often very deep and complex, molecular absorption bands are the main spectral characteristics of this component. All the bands are from oxides and include TiO, VO, AlO, ScO, and YO. The excitation temperature derived from the bands ranges from $\sim2500$ K, identified also as the photospheric temperature of V838 Mon, down to $\sim500$ K, which presumably corresponds to outflowing matter at a few stellar radii. The most positive (heliocentric) radial velocity derived from the bands is $\sim58$ km s$^{-1}$, which we propose to be the radial velocity of V838 Mon itself. Several atomic lines, mostly resonance, display P-Cyg profiles. They provide evidence of an intense mass outflow with a typical velocity of $\sim150$ km s$^{-1}$. The blue part of the spectrum is dominated by the spectrum of the B3V companion. The values of $T_{\rm eff}$ and log $g$ derived from the spectrum agree well with those implied by the spectral type. The star is a rapid rotator ($V$ sin $i \simeq 250$ km s$^{-1}$) and its (heliocentric) radial velocity is $\sim40$ km s$^{-1}$. Numerous emission lines were identified, mainly in the blue part of the spectrum. They correspond predominantly to \[\] and exhibit the same profile, which can be fitted well with a Lorentzian profile. The lines are centred on a heliocentric radial velocity of $\sim13$ km s$^{-1}$ and have a FWHM of $\sim80$ km s$^{-1}$.

propose that the appearance of the \[\] emission lines and the eclipse-like event observed in November/December 2006 were unrelated, and that the eclipse-like event was caused by an eclipse in a binary system. We argue that both events were strongly related and produced by matter ejected from V838 Mon during the 2002 outburst and reaching the vicinity of the B3V companion (see also ).

In principle, one can consider that because of a certain mechanism (e.g., dissipation of mechanical energy), the outer parts of the present V838 Mon wind become excited, producing the observed \[\] emission lines. In this case, however, the radial velocity of the lines would be close to that of V838 Mon. This is certainlly not the case. The \[\] lines have a radial velocity of $13.3 \pm 0.7$ km s$^{-1}$ (Sect. 4.1.1 in Paper I), while the radial velocity of V838 Mon is at least $53$ km s$^{-1}$ (see Sect. ). In this case, it would also be difficult to explain the observed \[\] profiles.

All the observed characteristics of the \[\] emission lines can, however, be easily explained, if they are assumed to be produced by the matter ejected during the 2002 eruption that approaches the B3V companion. As shown in Sect. , the observed line profiles as well as their radial velocities can then be well accounted for. The continuous strengthening of the lines in 2005–2006 is also easy to understand: larger and larger amounts of matter become excited as it approaches the source of excitation. Excitation by radiation from the B3V companion then explains why the emission line spectrum is dominated by . In the spectrum of a B3V star, there are enough photons capable of ionizing species with an ionization potential of $\sim 7$ eV, e.g., Fe, Ni, while there are very few photons capable of ionizing the most abundant elements such as H, He or CNO. Finally, the idea easy explains the $\sim 70$ day eclipse-like event observed in November/December 2006 as an occultation by a dense cloud of the matter ejected in 2002 and now crossing the line of sight of the B3V companion. If this is the case, one can expect that the 2006 eclipse should be followed by similar events when other fragments of the 2002 ejecta cross the line of sight. Indeed, a few months later V838 Mon entered another eclipse-like event.[^1] We note that the latter would be, at least, difficult to explain as a phenomenon typical of an eclipsing binary system.

# The radial velocity of V838 Mon

Several molecules observed in the spectrum of V838 Mon reveal bands formed from high excitation-energy levels (see Sect. 5 in Paper I). They are usually indicative of a radial heliocentric velocity of $58\pm5$ km s$^{-1}$ (see also Sect. ). It is reasonable to assume that they arise very close to the photosphere and have a velocity close to the stellar one. Earlier determinations of the stellar velocity in and implied a value of 59–62 km s$^{-1}$, i.e., close to our result.

However, the SiO maser emission observed in V838 Mon is at a heliocentric velocity of 71 km s$^{-1}$. In the case of late-type stars, the SiO masers usually have a reliable measure of their stellar radial velocity.

There are two possible explanations of this discrepancy. If the SiO maser emission really originates at the stellar velocity then the estimate made in Paper I may indicate that we do not measure molecular bands deep enough that are to reach hydrostatic levels of the V838 Mon atmosphere, i.e., even the highest excitation bands are formed in already expanding layers. The other possibility is that the optical spectroscopic studies provide a correct estimate of the stellar velocity but that the SiO maser is not at the stellar velocity. We note that the radial velocity and width of the main component of the SiO maser are very close to those of the CO rotational lines observed in diffuse matter in the close vicinity of V838 Mon . This point certainly deserves further investigation.

In the present paper, we usually adopt the radial velocity derived from the SiO maser, i.e., 71 km s$^{-1}$, as the radial velocity of V838 Mon. One has, however, to keep in mind that the results from optical spectroscopy infer a value 10–15 km s$^{-1}$ lower.

# An analysis of the \[\] line profile

As discussed in Sect. , we assume that the \[\] emission lines are formed in the matter ejected by V838 Mon during its 2002 outburst, which is now approaching the B3V companion and ionized by the UV radiation of the companion. In this section, using simplified modelling, we show that the observed profiles of the \[\] lines can be easily explained by this scenario.

## The model

We consider a system of two stars, $S_1$ and $S_2$, with a separation $A$ (see Fig. ). We assume that the star $S_1$ is a source of a steady, spherically symmetric wind, that forms an expanding envelope with a $r^{-2}$ density distribution. We allow for the possibility that matter in the form of clumps or filaments fills a $\xi$ part of the envelope volume, i.e., $\xi \le 1$. The wind started a certain time ago, so the envelope has an outer boundary, $r_0$. The wind has not yet reached star $S_2$, so $r_0 < A$. Star $S_2$ is a source of ionizing radiation and emits $N_{\rm ph}$ ionizing photons isotropically per unit time and unit solid angle. The radiation produces an ionized region in the wind whose boundaries can be determined assuming ionization equillibrium.

We consider ionization along a given direction from star $S_2$ forming an angle $\alpha$ with the axis between the stars. A distance $x_2$ from star $S_2$, to which the wind matter is ionized, can be found from $$\label{ion_eq}
N_{\rm rec} = N_{\rm ph}\,{\rm d}\omega,$$ where $$\label{rec_eq}
N_{\rm rec} = \int_{x_1}^{x_2} \xi\,x^2\,{\rm d}\omega\,\alpha_{\rm rec}\,
              N_{\rm i}^2\,{\rm d}x$$ is a number of recombinations in a cone of solid angle d$\omega$ along the considered direction, $\alpha_{\rm rec}$ is a recombination coefficient, while $N_{\rm i} = N_0\,(A/r)^2$ is an ion number density (electron density is assumed to be equal to ion density). In the following, we assume that $A$ is a unit length, so all the $r$ and $x$ are expressed in terms of $A$, and $x$ is then related to $r$ and $\alpha$ via $$\label{x_eq}
  r^2 = 1 + x^2 - 2\,x\,\cos\alpha.$$ In Eq. (), $x_1$ is a distance from star $S_2$ to the wind outer boundary along the considered direction, given by $$\label{x1_eq}
x_1 = \cos \alpha - \sqrt{\cos^2\alpha - (1-r_0^2)}.$$ Equation () can be rewritten as $$\label{ion_eq2}
C_0 \equiv \frac{N_{\rm ph}}{A^3\,\xi\,N_0^2\,\alpha_{\rm rec}} =
    \int_{x_1}^{x_2} \frac{x^2}{r^4}\,{\rm d}x,$$ which, using Eq. (), can be evaluated to give $$\label{ion_eq3}
C_0 = I(x_2) - I(x_1),$$ where $$\label{i_eq}
I(x) \equiv \frac{x\,\cos 2\alpha - \cos\alpha}{2\,r^2\,\sin^2\alpha}
     + \frac{1}{2\,\sin^3\alpha}\,\arctan\frac{x-\cos\alpha}{\sin\alpha}.$$ Equation () can be solved numerically to obtain $x_2$. Along the separation axis, i.e., when $\alpha = 0$, $x_1 = 1 - r_0$, while Eq. () reduces to $$\label{i0_eq}
I(x) = \frac{1}{1-x} - \frac{1}{(1-x)^2} + \frac{1}{3(1-x)^3}.$$ Solving Eq. () for a grid of $\alpha$ values in the range $0 \le
\alpha \le \arcsin r_0$, allows us to obtain the shape and position of the ionization front in the wind. Matter is ionized between the wind outer boundary and the ionization front. We note that the star separation axis is a symmetry axis of the ionized region. There are two free parameters in the above problem, i.e., $r_0$ and $C_0$.

We assume that the ionized region is isothermal, so that the emission line coefficient varies as $N_i^2$, and that the intrinsic line profile is Gaussian characterized by a thermal and/or turbulent velocity, $V_{\rm t}$. Integrating the intrinsic line profile over the ionized region and taking into account the kinematic properties of the ionized wind, a final emission-line profile can be obtained. Apart from the above-mentioned parameters determining the ionization front, the resultant line profile depends on kinematic parameters of the wind and the stellar system, which are: the wind expansion velocity, $V_{\rm wind}$; the velocity of star $S_1$ (source of the wind) relative to the observer, $V_{\rm s}$; and the angle between the stars separation axis and the line of sight, $\alpha_{\rm s}$.

## Fitting the model to the observations

Some of the parameters in the above problem can be estimated from observations. As discussed in Sect. , we assume that close to the time of the November/December 2006 eclipse, matter ejected during the 2002 eruption of V838 Mon reached the vicinity of the B3V companion. Our observations were completed in October 2005, so we can estimate that $r_0 \simeq 0.75$. During the 2002 outburst of V838 Mon, expansion velocities observed reached $\sim 600$ km s$^{-1}$, although most of mass loss occurred at 150–400 km s$^{-1}$ . As discussed in , the most intense mass loss occurred in March 2002, which was observed as an expanding photosphere of velocity $\sim 270$ km s$^{-1}$. We therefore assume $V_{\rm wind} \simeq 250~{\rm km\,s}^{-1}$ in the present calculations. We also assume that the heliocentric radial velocity of V838 Mon is $V_{\rm s} = 71~{\rm km\,s}^{-1}$ (see Sect. ). Thus, there remain 3 free parameters, i.e., $C_0$, $\alpha_{\rm s}$, and $V_{\rm t}$, which can be obtained by fitting the model profile to the observed profile of the \[\] emission lines.

A fit of this kind is presented in Fig. . Points show the observed profile of the \[\] 4287 Å line. This is one of the strongest emission lines of \[\] in our spectrum. Its profile is free of blending with other lines and affected little by noise. The local continuum was subtracted and the flux was normalized to the peak value. (Note that the profile of \[\] 4287 Å  shown in Fig.  is practically the same as the mean profile derived from 7 \[\] lines displayed in Fig. 4 of Paper I.) The model profile was obtained with the following values of the free parameters: $C_0 = 0.35$, $\alpha_{\rm s} = 77\degr$, and $V_{\rm t} = 11.5~{\rm km\,s}^{-1}$. If $V_{\rm s} = 58~{\rm km\,s}^{-1}$ is assumed (as discussed in Sect. ), the results of the profile fitting remain unchanged, except $\alpha_{\rm s}$, which increases to $80\degr$.

We note that there is little ambiguity in determining the values of the above free parameters, because each of them affects different characteristics of the model line profile. The value of $C_0$ determines the size of the ionized region, so, for a fixed $V_{\rm wind}$, it affects the wings of the line profile (the higher the value of $C_0$, the more extended and stronger the wings). If $V_{\rm wind}$ and $V_{\rm s}$ are fixed, $\alpha_{\rm s}$ defines the position of the line peak. Finally, $V_{\rm t}$ determines the width of the line core.

## Discussion

The results of the above modelling of the \[\] profile allows us to estimate the parameters of the matter approaching the B3V companion. From the profile fitting, we have $C_0 = 0.35$. Assuming that the matter, expanding with velocity $V_{\rm wind} \simeq 250~{\rm km\,s}^{-1}$, reached the B3V companion in November or December of 2006, we derive a distance between V838 Mon and the B3V companion of $A \simeq 3.7 \times 10^{15}$ cm ($\sim 250$ AU). Integrating the spectrum of a standard B3V star above the ionization potential of iron, one obtains $N_{\rm ph} =
2.4 \times 10^{46}\,{\rm s}^{-1}$. The recombination coefficient of is $\alpha_{\rm rec} \simeq 2.5 \times 10^{-12}\,{\rm cm}^3\,{\rm s}^{-1}$ (at an electron temperature of $10^3 - 10^4$ K; see ). From Eq. (), one then finds that $N_0 \simeq 7 \times
10^5\,\xi^{-1/2}\,{\rm cm}^{-3}$, or $N_i \simeq 1.3 \times
10^6\,\xi^{-1/2}\,{\rm cm}^{-3}$ at the outer edge of the wind envelope, i.e., at $r_0$. We note that at an electron density $\ga 10^7\,{\rm cm}^{-3}$, the \[\] lines become collisionally de-excited. Assuming spherical symmetry, a mass-loss rate during the 2002 outburst can be estimated to be $$\dot{M}_{\rm wind} = \frac{4\,\pi\,A^2\,V_{\rm wind}\,\xi\,N_0\,1.4\,m_{\rm H}}
                      {N_{\rm Fe}/N_{\rm H}},$$ where $N_{\rm Fe}/N_{\rm H}$ is the relative-to-H number density of elements with an ionization potential similar to or lower than that of Fe (i.e., Na, Mg, Al, Si, Ca, Ni), while $m_{\rm H}$ is the H atom mass. Assuming solar abundances, $N_{\rm Fe}/N_{\rm H} \simeq 10^{-4}$ and $\dot{M}_{\rm wind} \simeq 7 \times 10^{25}\,\xi^{1/2}\,{\rm g\,s}^{-1}
\simeq 1\,\xi^{1/2}\,{\rm M}_{\odot}\,{\rm yr}^{-1}$. The most intense period of mass loss lasted a month (March) in 2002 , so, according to the above estimate, it produced a $\sim 0.1\,\xi^{1/2}\,{\rm M}_{\odot}$ shell. This agrees with the total mass lost in the 2002 outburst, which was estimated to have a mass between $0.001 - 0.6\,{\rm M}_{\odot}$ in .

# Mass outflow in V838 Mon

noted presence of continuous mass loss since the outburst. They reported that the outflow observed in 7698 Å has not much changed since 2002. The nature of its mass loss is however unclear. In this section, we analyse and discuss results from the profiles of molecular bands and atomic lines observed in October 2005 and presented in Paper I.

## Outflow as seen in molecular bands

As discussed in Paper I, the spectrum of V838 Mon is dominated by numerous, complex, and often very deep absorption lines related to molecular bands. A simple model of a stellar photosphere plus an outflowing homogeneous layer was used to identify and fit the observed structures of individual bands. In this procedure, it was necessary to adopt various velocities for the outflowing layer to reproduce different band structures. In general, we found that bands originating in more excited levels correspond to higher (more positive) radial velocities than low excitation bands. This effect is shown, in a more quantitative way, in Fig. , which plots the heliocentric radial velocity of a particular band versus the excitation energy, $\chi$, of the level from which the band arises. Only results for the TiO molecule, for which we identified the largest number of bands, are shown. As the excitation potential plotted in Fig. , we have taken the energy of the lower level of the most blueshifted line in the bandhead. Vertical error bars illustrate uncertainties in the velocity. Uncertainties in the energy have been taken as a range of energies of lower levels of lines lying within $\pm35$ km s$^{-1}$ of the bandhead (this is the typical velocity broadening adopted in modelling the bands in Paper I).

As can be seen from Fig. , there is a clear correlation between the radial velocity of a particular band and the excitation potential of the lower level of the band. This we interpret as evidence of an outward accelerated mass outflow, in which the observed bands are formed. The highest excitation bands can be formed in hotter regions, i.e., close to the photosphere, where the outflow velocity is low. Hence, we postulated in Paper I that the highest radial velocity observed in the high excitation bands, i.e., $\sim +58$ km s$^{-1}$, corresponds to the radial velocity of V838 Mon. The bands originating in levels close to the ground state can be easily formed in cold outermost layers outflowing with a large velocity.

Figure shows the same data as in Fig.  but in a different, more astrophysical, way. Here, instead of the excitation potential, we plot a parameter, log$(gf\lambda)-5040\chi/T_{\rm exc}$, where $gf$ and $\lambda$ are the oscillator strength and wavelength of the band, respectively. Within a constant factor, this parameter is equivalent to the logarithm of the absorption coefficient in a given band. As an excitation temperature, $T_{\rm exc}$, we adopted 500 K, which is a typical value in the outflow, as inferred in Paper I (Sect. 5). The horizontal error bars show uncertainties related to the spread in the excitation energy and oscillator strength for the lines present within 35 km s$^{-1}$ of the bandhead.

As Fig. , Fig.  can be interpreted as evidence of an increasing outflow velocity with the distance from the photosphere. Bands with a small absorption coefficient require large column densities to be visible in the observed spectrum. Thus, they must be formed deep in the outflow. More opaque bands are effectively formed in more distant layers and their profiles provide information about the kinematics well above the photosphere.

From the span of the radial velocities observed in the TiO bands, shown in Figs.  and , we can conclude that the wind in V838 Mon is accelerated at least to a velocity of $\sim 180$ km s$^{-1}$.

## An analysis of the P-Cyg profiles

The presence of lines showing P-Cyg profiles in the spectrum of an observed object is usually considered as strong evidence of an ongoing mass outflow from the object. In our spectrum of V838 Mon, we found about ten atomic lines exhibiting these characteristics (see Sect. 4.2 in Paper I). All of them but one originate from resonant transitions. Below we analyse the profiles of these lines in an attempt to investigate the physics and nature of the outflow.

The terminal velocity of the outflow from V838 Mon can be obtained from the observed  $\lambda$`<!-- -->`{=html}7698 absorption profile (see Fig. 6 in Paper I). If we take V$_s = 71$ km s$^{-1}$ to be the stellar radial velocity (see discussion in Sect. ), then the terminal velocity of the outflow is V$_{\infty} = 215 \pm 5$ km s$^{-1}$. The observed absorption components in other P-Cyg profiles do not reach this velocity, most probably because of the low column density of the atoms in the outer parts of the outflow. Because of the high oscillator strength and the high atomic abundance, the opacity in the  $\lambda$`<!-- -->`{=html}7698 line is at least two orders of magnitude higher than for any other resonant line observed. Consequently, this line can be observed even with relatively low column densities expected at high velocities. The value of terminal velocity derived from the absorption is consistent with, although slightly higher than, the outflow velocity estimated form the molecular bands of TiO analysed in Sect. . It also agrees with the maximum expansion velocity derived by from the CO bands observed half a year after our observations.

To obtain a deeper insight into the nature of the outflow in V838 Mon, we performed a radiative transfer modelling of the P-Cyg profiles. We assume that the profiles are formed in a spherically symmetric expanding envelope. Our modelling procedure is based on the SEI method developed by , and all details concerning the solution of radiative transfer can be found therein[^2]. As the velocity field of the outflow, we adopt the standard $\beta$-law, i.e., $$\label{betalaw}
w(x)=w_0+(1-w_0)\left(1-\frac{1}{x}\right)^{\beta},$$ where $w=V/V_{\infty}$ is an outflow velocity normalized to the terminal velocity, $x=r/{R_{\star}}$ is a dimensionless radial distance from the star with a photospheric radius $R_{\star}$, while $w_0$ is the normalized initial velocity at the stellar photosphere (taken to be $w_0$=0.01 in our calculations).

As already pointed out in Paper I (Sect. 4.2), the observed P-Cyg absorption components have velocity structure. A majority of the absorption profiles have narrow absorption components (NACs) that appear at different velocities. The most striking example of NAC is seen in the line at V$_h$ = –82 km s$^{-1}$(see Fig. 6 in Paper I). Some lines, e.g., those of and , seem to exhibit many NACs at once, which dominate the appearance of the profiles. On the other hand, there are also absorption profiles, e.g., those of $\lambda$`<!-- -->`{=html}6572 and $\lambda$`<!-- -->`{=html}7698, which are very smooth and do not seem to be affected by discrete components. The question is whether the narrow components seen in our profiles are superimposed on a broad P-Cyg absorption profile formed in a bulk flow, or whether the entire absorption profile consists of several discrete and strong absorption components that are blended and together form a more or less continuous profile.

As a first attempt, we assumed a distribution of optical thickness in the form $$\label{tau1}
\tau(w)={\mathcal T}_{\rm tot}\,(w/w_1)^{\alpha_1} \left(1-
        (w/w_1)^{1/\beta}\right)^{\alpha_2},$$ where ${\mathcal T}_{\rm tot}=\int_{w_0}^{w_1}\tau(w)\,dw$ is the integrated optical thickness and $w_1$ is the normalized velocity at which opacity vanishes (usually taken to be 1). The values of ${\mathcal
  T}_{\rm tot}$, $w_1$, $\alpha_1$, and $\alpha_2$ were treated as free parameters in our modelling procedure. The radial distribution given by Eq. () is known to reproduce ultraviolet (UV) resonant lines formed in winds of hot stars . Our attempts to reproduce the smooth profiles of , , and the overall profile of (neglecting the NAC), showed that it was impossible to reproduce satisfactorily any of these lines with the opacity distribution in the form of Eq. (). The best set of the parameters found within this approach resulted in profiles that were too narrow with respect to the observed ones.

We, therefore, modified the optical depth distribution by adding multiple Gaussian components, $\tau_i(w)$, to Eq. () parameterized by central velocities, widths, and weights according to which they are added. We note that, for the sake of convenience, the Gaussian components are parameterized in the velocity domain, so the corresponding distributions as functions of radial distance, $\tau_i(x)$, are in general not Gaussian (the profile is modified by the adopted velocity field).

In this multi-component approach, we were able to successfully reproduce the observed profiles. Results for the $\lambda$`<!-- -->`{=html}7800 line are shown in Fig. . At least three discrete components ("shells") of different widths are needed to obtain a profile that reproduces the observations well. These components, which are clearly present in all the profiles, are located at outflow velocities of about 50, 110, and 150 km s$^{-1}$ (with respect to V$_s$ = 71 km s$^{-1}$).

We attempted to model all the prominent P-Cyg lines independently and, while the central velocities and widths of different components were found to be consistently determined for different lines, the relative contributions of the components differ considerably from line to line. This implies that the relative ion opacities differ considerably between consecutive "shells". This can be understand either as changes in excitation conditions or different atomic abundances (chemical composition) in the discrete outflow components. Although the latter seems to be less likely, the changing atomic fractions due to interchanges of atoms with molecules and/or dust grains may play an important role. An extreme case of this effect is observed in the resonant absorption lines of . Apart from the main component seen close to the stellar velocity (but see Sect. ), these lines appear only in one of the outflowing "shells" (see Fig. 6 in Paper I).

Our modelling allows us to place general constraints on the velocity field in the outflow. The adopted value of $\beta$ has a strong influence on the relative strengths of absorption and emission components in a P-Cyg profile (for a given radial distribution of optical thickness). We found that the outflow in V838 Mon can be characterized well by $\beta=$`<!-- -->`{=html}0.35$\pm$`<!-- -->`{=html}0.05. This result was obtained assuming that collisional deexcitation and photospheric contamination to the P-Cyg profile can be neglected. An influence of collisional de-excitation on the emission component cannot be properly recognized because of unknown electron densities in the outflow. We believe that it is small, at least for lines with high transition probabilities (e.g., the lines of , , and with an Einstein coefficient A$_{ul}$ of the order of 10$^7$–10$^8$ s$^{-1}$). We did not find any signs of photospheric contamination in the observed spectrum and we therefore neglected this effect in our modelling.

The value of $\beta$ derived above implies that the acceleration of material is very rapid and at the distance of 4$R_{\star}$, the wind has a velocity of 0.99 $V_{\infty}$. This steep velocity gradient requires very high densities of material to be able to reproduce strong absorption features at low velocities. Therefore, to reproduce the absorption component at the outflow velocity of 50 km s$^{-1}$ (which corresponds to r = 1.03 $R_{\star}$ for $\beta=0.4$), it was necessary to include an inner narrow shell with a very high optical thickness. For several lines, e.g., the line, it was even impossible to reproduce the central parts of the P-Cyg profile without incorporating unreasonably high values of optical thickness. This made us conclude that in the velocity field, generally well described by the $\beta$-law, a sort of a plateau (or plateaus) may exist, i.e., a region of nearly constant velocity extended in the radial direction. These velocity plateaus, or non-monotonicities, were proposed to explain the winds of hot stars . The possible existence of similar features in the outflow of V838 Mon is a tentative conclusion, which should be examined by time-dependent observations of the P-Cyg profiles in the spectrum of V838 Mon. The derived radial distributions of optical thickness for different ions can be used to provide a rough estimate of the mass-loss rate in V838 Mon. For a given stellar radius, $R_{\star}$, the mass-loss rate can be estimated from $$\label{mdot}
\dot{M}(x)=\tau(x)\,w(x)\,x^2\,\frac{dw(x)}{dx}\,V_{\infty}^2\,R_{\star}
     \,\frac{4\,m_{\rm e}\,c}{e^2}\,\frac{\mu\,m_{\rm H}}{A_E\,f_{lu}\,\lambda_0},$$ where $\mu$ is the mean molecular weight, $m_{\rm H}$ is the hydrogen mass, $A_E$ is the abundance of an element with respect to hydrogen, $f_{lu}$ is the oscillator strength, $\lambda_0$ is the laboratory wavelength of the line, and the symbols $m_{e}$, $e$, and $c$ have their usual meanings. In Eq. (), it is assumed that all the atoms of a given element are in the ground state and that there is no ionization nor any other form of ion depletion. Taking as the stellar radius $R_{\star}$ = 10$^3$ R$_{\sun}$ (see Sect.5 in Paper I), adopting $\beta=0.4$, and assuming solar abundances, the mass-loss rate derived from the lines of $\lambda$`<!-- -->`{=html}7800 is on average of $10^{-6} - 10^{-5}$ M$_{\sun}$ yr$^{-1}$, with individual mass-loss events reaching $10^{-3}$ M$_{\sun}$ yr$^{-1}$ (see Fig. ).

## Signatures of an infall

The redward broad absorption components seen in the lines (see Sect. 4.3 in Paper I) can be interpreted as a signature of an infall in V838 Mon. Similar absorption components also contaminate the profiles of the 5060 and 5110 Å lines of .

These absorption features in the lines have wings extending out to about $V_h$ = 120 km s$^{-1}$. Thus, regardless of the velocity of the star in the range 58–71 km s$^{-1}$ (see Sect. ), the red wings of these features are evidently redshifted with respect to the photosphere. The maximum infall velocity inferred from these wings is 50–60 km s$^{-1}$, which agrees very well with the free fall velocity of an $8 M_\odot$ star at a photospheric radius of $\sim 900 R_\odot$ (Sect. 5 in Paper I).

Signatures of an infall were previously observed in V838 Mon. observed inverse P-Cyg profiles of lines in their spectra obtained in the near infrared in December 2003. From the separation of the absorption and emission components, an infall velocity of $\sim 20$ km s$^{-1}$ can be deduced. These authors also argue that the infalling matter, compressed, and heated when colliding with the atmospheric material, provide conditions necessary to excite the SiO lines that they observe. Observations of the ro-vibrational bands of CO by in April 2006 revealed a photospheric component redshifted by 15 km s$^{-1}$ with respect to the velocity of the SiO maser. This suggests, according to the authors, the presence of gas infalling onto the star or contraction of the stellar photosphere.

## Discussion

As mentioned above, the mass outflow in V838 Mon has been observed continously since the 2002 eruption. Our analysis confirms this statement and provides new results. The question that arises is what drives this outflow. It consists of cold neutral matter that is rich in molecules, and possibly also dust, which is similar to the composition of the winds of cool stars, e.g., objects in the asymptotic giant branch. Velocity fields of these cool winds are usually well described by the $\beta$-law with $\beta$ $\simeq$ 0.5, which is close to the value derived above for V838 Mon. These cool winds are believed to be dust-driven. They usually do not reach the high terminal velocities found for V838 Mon. A possible reason is that dust is destroyed in collisions, when the outflow is faster than about 30 km s$^{-1}$ . From this reason, it seems that the outflow in V838 Mon cannot be driven by dust only, since we observe material being accelerated at much higher velocities than 30 km s$^{-1}$. This driving mechanism can, however, play a role at the base of the outflow, where velocities are low enough.

One can consider a mechanism similar to that driving winds in hot stars, i.e., by radiation pressure absorbed in atomic resonant lines. However, the small number and relative weakness of lines of this type in our spectrum (they are typically 10 times narrower than those in the OB stars) does not support the idea that these lines could be effective in driving mass loss from V838 Mon. More promising is the possibility that the outflow is driven by radiation pressure absorbed in molecular bands in the optical and infrared. The object is very bright in these wavelength ranges and the observed number and strength of molecular bands, as seen in our spectrum (some of them absorb practically all the radiation available in their wavelength ranges – see Paper I), clearly shows that a significant part of the momentum carried out in radiation is indeed absorbed in the outflowing matter. A process of driving winds by radiation pressure absorbed in molecular bands was proposed as a mechanism accelerating cold circumstellar envelopes .

The analysis of the P-Cyg profiles reported in Sect.  was completed assuming spherical symmetry and a monotonicly increasing outflow velocity described by Eq. (). Within this approach, the general shape of the observed profiles can be satisfactorily reproduced. This shows that the general pattern of the outflow is more or less spherically symmetric. The profiles however show, at least some of them, narrow structures, which we called NACs. The structure at $V_{\rm h} = -82$ km s$^{-1}$ is particularly striking. In the spherically symmetric, approach the only way to explain the NACs is to assume that the mass-loss rate varies with time and produces dense shells expanding with a thinner wind. As can be seen in Fig. , to explain the strong NAC at $V_{\rm h} = -82$ km s$^{-1}$ it is necessary to postulate a short lived enhancement in the mass loss rate by two orders of magnitude. However, if the above assumptions are relaxed, the NACs can be explained in other ways. One possibility is that the NACs are produced by mass loss being more intense in certain, more or less discrete, directions. For instance, the NAC at $V_{\rm h} = -82$ km s$^{-1}$ would then be understood as a stream-like or jet-like outflow with a projected velocity of $\sim 153$ km s$^{-1}$ (adopting the radial velocity of V838 Mon of 71 km s$^{-1}$; see Sect. ). Jets usually have counter-jets. The observed lack of any counter-jet in our spectrum as well as the position of the NAC close to the terminal velocity of the P-Cyg profiles suggests that the jet, if responsible for producing the NAC at $-82$ km s$^{-1}$, is not far from the line of sight.

Another possibility is that the NAC at $V_{\rm h} = -82$ km s$^{-1}$ has nothing to do with the presently ongoing mass outflow, but is produced in the matter lost during the 2002 eruption that is now in front of the object. This interpretation would imply the existence of a dense shell at a distance of $\sim 1.7\,\times\,10^{15}$ cm ($\sim 2.5\,\times\,10^4\,\rm{R}_\odot$) from the object expanding with a velocity of $\sim 153$ km s$^{-1}$. However, during the 2002 eruption the matter was ejected with velocities ranging from $\sim 100$ to $\sim 600$ km s$^{-1}$ . Thus, the matter, or at least part of it, would have to have been condensed into a shell at later epochs. This can occur if matter ejected later catches up with slower matter ejected earlier. In the case of V838 Mon, this mechanism does not seem to have worked, since the outflow velocity observed during the 2002 eruption tended to decrease with time . A NAC at $V_{\rm h} = -90$ km s$^{-1}$ is seen in our spectrum in the  5147 and 5152 Å lines, which do not display P-Cyg profiles (see panel c in Fig. 6 in Paper I). The close radial velocity and line width suggest that this component is produced in the same enviroment as the NAC in the P-Cyg profiles. If this is the case, our interpretation presented above would have to be excluded. The lines arise from the levels lying 2.3–2.4 eV above the ground state. Thus, they must arise in a rather warm matter (of temperature comparable to that in the photosphere), while the matter lost in 2002 and now at a distance of $\sim 30$ photospheric radii is expected to be cold ($\sim 500$ K, if estimated from pure geometric dilution of the radiative energy density). In conclusion, we consider that the interpretation that the NAC at $V_{\rm h} = -82$ km s$^{-1}$ is related to matter lost in 2002 is less probable than the other two, i.e., varying mass-loss rate in the present wind or jet-like ouflow.

Spectroscopic data allowing analyses of the time-dependent behaviour of the P-Cyg profiles would help to discriminate between interpretations involving varying mass-loss rate, jet-like outflow, or distant matter lost in 2002. In the first case, the NACs are expected to migrate in the profile with time, while in the two other cases the NACs should be rather stable features. It is worth noting that in their high-resolution spectra obtained in the $K$ band in April 2006, i.e., half a year after our spectroscopy, indentified several velocity components in the CO absorption line profiles. Apart from the photospheric component (mentioned in Sect ) the authors found components at expansion velocities of 15, 85, and 150 km s$^{-1}$. The last feature is practically at the same velocity as our NAC at $V_{\rm h} = -82$ km s$^{-1}$ suggesting that this is a persistent component. However, the feature in the CO lines is much wider than the NAC, so the conclusion is unclear. The two other components in the CO lines do not match other NACs in our P-Cyg profiles, which are seen at the expansion velocities of 50 and 110 km s$^{-1}$ (see Sect. ). This suggests that these NACs are transient and produced by varying mass-loss rates.

# Summary and discussion

Our analysis of both the positions of the molecular bands (Sect. ) and the P-Cyg profiles (Sect. ) have shown that V838 Mon loses matter with a terminal velocity of $\sim 215$ km s$^{-1}$ at a rate of $10^{-6} - 10^{-5}$ M$_{\sun}$ yr$^{-1}$. Thus, the object loses of the order of 10 $L_\odot$ in the form of the kinetic energy of the wind. This is small compared to the radiation luminosity of the object, which is $\sim 2.4 \times 10^4~L_\odot$ (see Sect. 5 in Paper I). However, the momentum carried out in the wind is comparable to that in the radiation. Thus, the wind in V838 Mon is likely to be radiation driven, probably due to absorption in the molecular bands, as discussed in Sect. .

The kinematic pattern of the matter in the atmosphere of V838 Mon is very complex. Both expansion and wind outflow dominate. Most of the outflow is more or less spherically symmetric as suggested by our modelling of the observed P-Cyg profiles (Sect. ). However, as discussed in Sect. , it is quite possible that we also observe a jet-like outflow. As shown in Sect. , certain regions, perhaps initially accelerated but not sufficiently to allow them to leave the object, also fall back to the photosphere. We note that simultaneous signatures of infall and outflow have been observed in the winds of hot stars and often in protostars. A possible scenario for creating coexisting mass outflow and infall in asymptotic giant branch stars was discussed in and may also work in the remnant of the V838 Mon outburst.

As discussed in , the evolution of V838 Mon in the post-outburst state is probably dominated by gravitational contraction, i.e., is closely related to that of protostars. If, as argued in , the 2002 outburst of V838 Mon was produced by a stellar merger, then a significant amount of angular momentum should have been stored in the remnant of the event . Gravitational contraction may then result in the flattening of the envelope, eventually leading to the formation of an accretion disc (as in protostellar objects). The possibility discussed in Sect. , namely that we observe a jet in the spectrum, if confirmed, may imply that a disc has already formed in V838 Mon.

As shown in Sect. , the observed profile of the \[\] lines can be easily understood as having been produced by matter lost by V838 Mon in the 2002 eruption and now excited by the radiation of the B3V companion. As the matter approaches the companion, a larger and larger proportion of the flux from the star can interact with the matter causing an increase in the \[\] emission, as observed. Indeed, the flux in the 20 strongest \[\] lines in our spectrum obtained in October 2005 increased by a factor of $\sim 3.5$ in the measurements completed by in December 2006. At the moment of our observations, the outer boundary of the expanding envelope was presumably at a distance of $\sim 0.75$ times the separation between V838 Mon and its B3V companion. Thus, only $\sim 17$% of the radiation flux of the B3V companion should then interact with the matter. At the moment of the eclipse-like event in November/December 2006, the matter presumably reached the closest vicinity of the companion and the portion of the stellar radiation capable of exciting the matter increased to $50-100$%.

Apart from the \[\] emission lines, also observed strong Balmer lines in emission. In particular, they discussed in detail the profile of H$\alpha$ observed in December 2006 – April 2007. No H$\alpha$ was seen in our spectrum in October 2005. However, in the absorption cores of higher members of the Balmer series (in the spectrum of the B3V companion), we have detected weak emission components (see Sect. 4.1.2 in Paper I). In particular, the emission component in the H$\beta$ line was estimated to have an observed flux of $\sim 3 \times
10^{-16}$ erg cm$^{-2}$ s$^{-1}$. We note that the profile of the H$\beta$ emission component (see Fig. 5 in Paper I) was remarkably similar to that of the H$\alpha$ line observed by in December 2006. Moreover, the parameters of the emission component in H$\beta$ in our spectrum and of the H$\alpha$ emission component in (radial velocity and FWHM) are close to those of the \[\] lines derived in Paper I. This implies that all these lines probably originate in the same region, i.e., in the part of the V838 Mon envelope (ejected in 2002) approaching the B3V companion. The point is, however, that while in the spectrum of a B3V star there are enough photons capable of ionizing iron and exciting the \[\] lines, there are very few photons, capable of ionizing hydrogen. Between October 2005 and December 2006, the flux in the H$\beta$ emission also increased by a factor of $\sim 7$, while that in H$\alpha$ increased by at least two orders of magnitude. This large increase cannot be explained by a purely geometric effect, as discussed above in the case of the \[\] lines. Therefore, an additional source of excitation of the Balmer lines is necessary, a source, whose intensity significantly increased between October 2006 and December 2007. We suggest that this is provided by matter accretion from the outflowing V838 Mon envelope on the B3V companion. We now complete simplified estimates for this scenario.

We assume that matter flows with a velocity, $V_{\rm wind}$, in the vicinity of a star of mass, $M_{\rm B3V}$. An accretion radius, $r_{\rm acc}$, can be estimated from the standard formula $$\label{racc}
  r_{\rm acc} \simeq \frac{2\,G\,M_{\rm B3V}}{V_{\rm wind}^2}
              \simeq 3 \times 10^{12}\,{\rm cm}
              \simeq 43\,R_\odot,$$ where we adopted $M_{\rm B3V} = 7\,M_\odot$ and $V_{\rm wind} =
250$ km s$^{-1}$. An accretion rate can then be estimated from $$\label{macc}
 \dot{M}_{\rm acc}
     \simeq \frac{\pi\,r_{\rm acc}^2}{(4\,\pi\,A^2)}\,\dot{M}_{\rm wind}
     \simeq 1 \times 10^{19}\,\xi^{1/2}\,{\rm g\,s}^{-1},$$ where we used the values of $\dot{M}_{\rm wind}$ and $A$ estimated in Sect. . This implies an accretion luminosity, $$L_{\rm acc} = \frac{G\,M_{\rm B3V}\,\dot{M}_{\rm acc}}{R_{\rm B3V}}
    \simeq 9\,\xi^{1/2}\,L_\odot,$$ where $R_{\rm B3V} = 4.5\,R_\odot$ has been assumed. This luminosity is small compared to the intrinsic luminosity of a standard B3V star ($\sim 1.9 \times 10^3\,L_\odot$, ). Thus, no significant effect is expected to be observed in the global energetics of the B3V companion, e.g., in the visual brightness. However, after being dissipated via an accretion shock[^3] and thus producing a hot layer above the star surface, the accretion luminosity is expected to be emitted mainly in the far-UV and/or soft X-rays. This can affect the ionization of the expanding envelope of V838 Mon, i.e., produce significant ionization of hydrogen. The accretion from the expanding matter is expected to become effective just before and during the November/December 2006 eclipse. We suggest that this is the reason for the rapid increase in emission of the Balmer lines, particularly in H$\alpha$, observed in the second half of 2006. We note that the luminosity in the Balmer emission lines, dominated by H$\alpha$, as observed by during the eclipse, was $\sim 3.5\,L_\odot$ (assuming $E_{B-V} = 0.9$ and a 8 kpc distance to V838 Mon). Thus, the accretion, as estimated above, can provide enough energy to explain the observed luminosity in the Balmer lines.

[^1]: See e.g., the website of V. Goranskij (`http://jet.sao.ru/`$\sim$`goray/v838mon.htm`).

[^2]: Aware of limitations of the SEI method we have also performed modelling in the more accurate comoving frame formalism . It confirms the results obtained with the former method.

[^3]: The specific angular momentum accreted with the matter is negligible in this case and there is no chance to create an accretion disc arround the B3V companion, contrary to the suggestion of .


abstract: MPJ Express is a messaging system that allows computational scientists to write and execute parallel Java applications on High Performance Computing (HPC) hardware. Despite its successful adoption in the Java HPC community, the MPJ Express software currently does not provide any support for debugging and profiling parallel applications and hence forces its users to rely on manual and tedious debugging/profiling methods. Support for such tools is essential to help application developers increase their overall productivity. To address this we have developed debugging and profiling tools for MPJ Express, which are the main topic of this paper. Key design goals for these tools include: 1) maintain compatibility with existing logging, debugging, and visualizing tools, 2) build these tools by extending existing debugging/profiling tools instead of reinventing the wheel. The first tool, named *MPJDebug*, builds on the open-source Eclipse Integrated Development Environment (IDE). It provides an Eclipse-based plugin developed using the Eclipse Plugin Development Environment (PDE). The default Eclipse debugger currently does not support debugging parallel applications running on a compute cluster. The second tool, named *MPJProf*, is a utility based on Tuning and Analysis Utility (TAU)—an open-source performance evaluation tool. Our goal here is to exploit TAU to profile Java applications parallelized using MPJ Express by generating profiles and traces, which can later be visualized using existing tools like `paraprof` and `Jumpshot`. Towards the end of the paper, we quantify the overhead of using MPJProf, which we found to be negligible in the profiling stage of parallel application development.
title: Design and Implementation of Parallel Debugger and Profiler for MPJ Express

National University of Sciences and Technology, Pakistan email@example.com National University of Sciences and Technology, Pakistan email@example.com National University of Sciences and Technology, Pakistan firstname.lastname@example.com

Design, Languages, Measurement

Java MPI Debugger, Java MPI Profiler

# Introduction

The Message Passing Interface (MPI) standard has become the *de facto* API for programming High Performance Computing (HPC) hardware including commodity clusters. The current version of the MPI standard supports traditional programming languages like C and Fortran by providing bindings for these languages. The two most popular implementations of MPI include MPICH and Open MPI . On the other hand, modern languages with features of object orientation, modularity, maintainability and portability have been treated with cynicism, mostly because of their poor computing performance and lack of high performance communication support . This criticism is not justified anymore because most modern languages and their compilers and runtime environments have witnessed manifold performance improvements. An example of one such modern programming language is Java. By the use of Just-in-Time (JIT) compilers the performance gap between Java byte code and native code is becoming negligible . The emergence of many popular and successful Java messaging libraries like mpiJava , FastMPJ and MPJ Express have successfully helped decrease communication gap between C/Fortran and Java applications on HPC hardware.

MPJ Express is an MPI-like—implements the mpiJava 1.2 API—messaging library with an active user community. The software is capable of executing in two modes named *cluster* and *multicore* modes. In the cluster mode, parallel applications execute in a typical cluster environment where multiple processing elements communicate with one another using a fast interconnect like Gigabit Ethernet or other proprietary networks like Myrinet and InfiniBand. In the multicore mode, the parallel Java application executes on a single system comprising of shared memory or multicore processors.

Despite its successful adoption in the Java HPC community, the MPJ Express software currently does not provide any support for debugging and profiling parallel applications and hence forces its users to rely on manual and tedious debugging/profiling methods, which require manually adding printing/logging/timing statements and constant recompilation of end user application. Support for such tools is essential to help application developers increase their overall productivity. In addition, manually debugging/profiling parallel applications is a complex, and challenging undertaking due to multitude of challenges including large scale parallelism, non-determinism, communication delays, synchronization requirements, concurrency control, and process locality.

To address this, we have developed debugging and profiling tools for MPJ Express named *MPJDebug* and *MPJProf* respectively. Key design goals for these tools include: 1) maintain compatibility with existing logging, debugging, and visualizing tools, 2) build debugging/profiling tools by extending existing debugging/profiling tools instead of reinventing the wheel. The first tool, named MPJDebug, builds on the open-source Eclipse Integrated Development Environment (IDE). MPJDebug is an Eclipse-based plugin—developed using the Eclipse Plugin Development Environment (PDE)—that allows MPJ Express users to execute and debug parallel Java applications running in the multicore mode or cluster mode. The default Eclipse debugger currently does not support debugging parallel applications running on a compute cluster. This is attractive because MPJ Express users can now utilize standard debugging features-including stepping, conditional and exception breakpoints, watch points and drop to frame—for their parallel Java applications. The second tool, named MPJProf, is a utility based on Tuning and Analysis Utility (TAU)—an open-source performance eval-uation tool. Our main goal is to provide MPJ Express users with a tool to analyze performance of their parallel Java applications. We achieve this by exploiting TAU to generate profiles and traces, which can later be visualized using existing tools like `paraprof`, `Jumpshot`, and `pprof`. Since we are building on a popular existing tool, different features and views are available for end users that include 3D visualization, threads-based and functions-based display.

Towards the end of the paper, we quantify the overhead of our profiling tool by employing variety of performance tests including basic latency and bandwidth benchmarks for point-to-point communication and Java NAS parallel benchmarks (NPB). Our results indicate that the MPJProf tool only adds a negligible overheads, which is due to generation of profiling information by parallel processes.

Rest of the paper is organized as follows. Section 2, discusses related work. Section 3 and 4 present implementation details of MPJDebug and MPJProf tools, respectively. This is followed by evaluating performance of the MPJ Express software with MPJProf in Section 5. Finally Section 6 concludes and discusses future work.

# Related Work

This section provides an overview of existing parallel debugging and profiling tools and also motivates the need for these tools in the context of our MPJ Express software. We begin our discussion with a review of debugging tools followed by profiling tools.

## Debugging Tools

*TotalView* is a powerful tool used for debugging parallel programs running on UNIX, Linux and Mac OS X. It supports the usual HPC application languages including C, C++ and Fortran. *Allinea Distributed Debugging Tool (DDT)* is a commercial software that is capable of debugging scalar, multithreaded and large-scale parallel applications. The tool supports C, C++, Fortran, Coarray Fortran, UPC, CUDA and OpenMP. *Eclipse* —an open-source IDE—features a built-in Java debugger that provides standard debugging features like breakpoint setting, step execution, suspend/resume threads and variable inspection. It is also capable of debugging remote applications. Eclipse IDE and debugger also support development/debugging in other popular languages including C, C++, and Python.

TotalView and DDT are the two most popular debugging tools for debugging C/C++/Fortran parallel MPI applications. Since these tools do not yet support Java, they cannot be used for debugging Java MPI applications parallelized with MPJ Express or FastMPJ. Being commercial tools, it is not possible for us to extend these tools. On the other hand, Eclipse being an open-source high-quality IDE is a good candidate to debug parallel Java MPI applications running in cluster and multicore modes. Our investigations suggest that it is straightforward to debug Java MPI applications running in the multicore mode. On other hand, the vanilla Eclipse debugger does not support debugging parallel Java applications running in the cluster mode—we add this feature in Eclipse as part of our main contribution in this paper.

## Profiling Tools

*JProfiler* is a commercial tool for profiling Java SE/SE applications. Salient features of the tool includes an intuitive Graphical User Interface (GUI) that allows users to find performance bottlenecks, pin down memory leaks, and resolve threading issues. JProfiler is capable of profiling a single Java Virtual Machine (JVM) application. However, an MPJ Express application typically consists of a group of Java processes—executing in separate JVMs. Another similar profiling suite is the *JProbe* software, which contains various tools to analyze performance of Java applications. These tools allow end-users to conduct post-mortem analysis of applications. JProbe is also a commercial product and it is thus not possible to extend it to support parallel MPJ Express programs.

*Tuning and Analysis Utility* (TAU) is a portable profiling and tracing toolkit for performance analysis of parallel programs written in Fortran, C/C++, UPC, Java and Python. TAU provides graphical and command line tools such as `paraprof` and `pprof` to visualize profiling results in nodes/threads and aggregated format. Typically users utilize open-source and free tools like `Vampir`, `Jumpshot`, and `Paraver` to visualize event traces. TAU also support some well-known parallel programming implementations like MPICH, Open MPI and mpiJava. As part of this paper, one of the objectives is to exploit TAU to profile Java applications parallelized using MPJ Express.

# Implementation of MPJDebug

This section details the implementation of our debugging tool for MPJ Express called MPJDebug. The first sub-section presents requirements from the perspective of our users. This is followed by an overview of the Eclipse plugin development architecture. Finally, we layout implementation details of our plugin that allow us to *launch* or *debug* parallel Java applications.

## Requirements of Debugger

In the context of MPJ Express users, the MPJDebug tool must support the following requirements:

1.  *Basic features*: Provide basic debugging features such as stepping, breakpoints and suspend/resume.

2.  *Ease to use*: Must be be usable—in terms of ease of use—by the application developers.

3.  *Scalable*: Must be scalable enough to allow debugging large parallel programs executing on hundreds and thousands of processors.

4.  *Remote debugging*: Must support MPJ Express multicore and cluster modes. When executing in cluster mode, a debugger is required to be able to connect to remote parallel processes running on cluster nodes.

With these goals in mind, a plausible approach to develop a debugger for MPJ Express is to extend a sequential debugger, like provided by Eclipse IDE, for compute clusters .

## Eclipse Plugin Architecture

The Eclipse Platform is an open framework, which consists of core technologies like Java Development Tools (JDT) and Plugin Development Environment (PDE) as shown in Figure  . The core platform consists of some essential components including a platform runtime—again depicted in Figure  . The functionality of the core Eclipse platform can be extending by building new plugins using the Plugin Development Kit (PDK) alongwith JDT and PDE. Owing to its modular architecture, many essential development tools are provided by Eclipse community as plugins. These plugins interact with one another and the core Eclipse platform using standard and published interfaces called *Extension Points*—these are depicted by power sockets in Figure  .

We follow the same approach for building the MPJDebug tool. Each plugin is developed as a self-contained software module, which contains the plugin manifest file—named `plugin.xml`. This file is written in the XML format and is typically load first by the Eclipse platform to customize the new plugin. The manifest file contains all necessary configuration information including details of extension points and display items like icons/menu items.

Eclipse workbench cannot launch or debug applications on its own. Launching means executing a program without it being suspended or examined and debugging means, execution may be suspended and resumed, variables may be inspected, and expressions may be evaluated. Eclipse workbench uses different set of plugins to acquire this feature. Debugger is one such plugin that Eclipse uses to launch or debug applications. Client/server design of debugger can debug programs running on local machines and running remotely on other systems in the network. The debug client runs on local workstation and debugger server runs on the same system as the program you want to debug. This could be a program launched on local workstation (local debugging) or a program started on a computer that is accessible through network (remote debugging). Left side of Figure   display local debugging where both debuggee process/program and debugger client are at the same machine and remote debugging is shown in the right side of   where debuggee process/program is at remote machine while debugger client is running inside the workbench on local machine and both machines are connected through network. Eclipse has a special Debug view that displays stack frame for suspended threads for each target you are debugging.

## Implementation of launching feature

Implementation of MPJDebug is based on Eclipse plugin architecture. It is developed by extending various features of Eclipse debugger to MPJDebug. To gain launching feature, MPJDebug extends Eclipse debugger feature of launching . Implementation of launching feature in MPJDebug is breakdown in different steps. Each step is described below.

**Step-1: Extension of LaunchConfigurationType**: Eclipse debugger uses `LaunchConfigurationTypes` extension point to launch Java applications. MPJDebug uses this extension point to achieve launching feature. This extension point provides a configurable mechanism for launching applications. Each launch configuration type has a name, supports one or more modes (run and/or debug), and specifies a delegate responsible for the implementation of launching an application . In Eclipse debugger, we have "Java Application" and "Remote Java Application" configuration types. This extension is declared in plugin manifest file. Code snippet for that extension is provided below

    <extension point="org.eclipse.debug.core.launchConfigurationTypes">
        <launchConfigurationType
            name="MPJExpress Application"
            delegate="JavaLaunchDelegate"
            modes="run,debug"
            id="mpjExpress">
        </launchConfigurationType>
    </extension>

**Step-2: Declaration of delegate**: Delegate attribute of `LaunchConfigurationType` is the most important one as it specifies fully qualified name of class that implements the interface `ILaunchConfigurationDelegate`. Second step is to create delegate class that implement interface `ILaunchConfigurationDelegate`. This class has a method `launch` which takes all required information through parameters and launches one of the modes defined in configuration type . The `launch` method is the first function that is invoked when an application is executed in new configuration.

    launch(ILaunchConfiguration configuration,
                String mode, ILaunch launch,
                IProgressMonitor monitor)
        throws CoreException

**Step-3: Definition of launch method**: Launch method takes different parameters of which `ILaunchConfiguration` parameter is important one as it contains all pertinent information that is related to launching of an application such as program arguments and VM arguments. In case of launching of MPJ Express application this configuration parameter contains information like number of processes, path to root directory of MPJ Express, device type and any additional parameters. All of this information is provided to configuration through MPJ parameters tab.

**Step-4: Implementation of MPJ parameters tab**: This tab is implemented using extension `launchConfigurationTabGroup`. This tab acts as graphical user interface where user can provide different options to launch parallel Java application. Options available in this tab are name of device, number of processes, path to MPJ Express root directory and support for different parameters. MPJ parameters tab is added as a part of `launchConfigurationTabGroup` so options from other tabs such as program arguments and VM arguments are available to the users. Figure   depicts MPJ parameters tab is configured to launch parallel Java application in multicore mode MPJ Express.

**Step-5: Final Launching**: After all pertinent information is provided in MPJ parameters tab and application is launched in run mode, `launch` method of delegate class is invoked. In `launch` method, information contained in configuration parameter is extracted and is modified to form a command that is used for launching MPJ Express applications. That command is then appended to VM argument of configuration. Finally configuration is launched using `IVMRunner`.

## Implementation of debugging feature

`LaunchConfigurationTypes` can execute Java applications in debug mode as well. To debug Java applications we need to enable debugger agent by passing following options in VM arguments of configuration.

    -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000

Parameter `address` acts as transport address for the socket connection between debugger client and debuggee process (from now on MPJ process). Java Debug Wire Protocol (JDWP) is used as communication protocol between MPJ process and debugger client. If `server=y`, then MPJ process is launched in debug mode and listen on port (specified in transport address) for debugger client to connect to it. These debug options are shifted to runtime system of MPJ Express where transport address for each process is set. MPJDebug is capable of debugging parallel Java applications in following two modes.

**Local Debugging** means both MPJ process being debugged and debugger client are at the same workstation. Debugging of parallel Java applications running in multicore mode of MPJ Express come under this category. To launch parallel Java applications in debug mode we provide a debug parameter with port as value through MPJ parameters tab. Value of this parameter is passed to runtime system of MPJ Express where it is used as value for address in debug options. MPJ process starts listening on set port. Debugger then connects to the listening port and application is launched in debug mode where different features can be used to debug application. Local debugging by MPJDebug can be seen in left side of Figure   where MPJ process and MPJDebug are at the same machine and port assigned to MPJ process is 8000.

**Remote Debugging** means MPJ process being debugged is at some remote machine (accessible through network) and debugger client is at local workstation. Debugging of parallel Java application running in cluster mode of MPJ Express come under this category. MPJ processes are distributed across the nodes of cluster, and there is possibility of more than one MPJ process will launch at one node. So a different port for each MPJ process launched at one node of cluster is required otherwise debugger throws an error stating "address already in use". Same port is used by MPJ processes launched at different nodes. To make sure different port is assigned to each MPJ processes at one node, we use the value provided against debug parameter in MPJ parameters tab and set a different address for each MPJ process. Formula used to generate new address is `(initial_debug_port + (2 * n))` where 'n' ranges from 0 to maximum number of MPJ processes that are to be launched at one node. If user provides `initial_debug_port` as 8000 and number of nodes are 2 then each node will host 2 processes and following ports will be set for each process. Right side of Figure   is an illustration of this example.

    Node-0 Process-0:8000; Node-1 Process-2:8000
    Node-0 Process-1:8002; Node-1 Process-3:8002

At the same time we are writing these ports along with names or IP addresses of nodes in a configuration file called `mpjdev.conf`. This file is accessible to each compute node and MPJDebug. It contains information of each compute node including IP address, process rank and debug port. MPJ Express runtime system use this file to assign debug port to each process. Once port assigning is completed, MPJ processes are launched and start listening on respective port.

MPJDebug reads `mpjdev.conf` file and retrieve IP addresses and their respective ports. Finally Java VM Connector is used to establish a connection to remote processes and JDWP communicate with MPJ processes using their respective port. Application can be further debugged using different debugging features.

# Implementation of MPJProf

This section details the implementation of our profiling tool for MPJ Express called MPJProf. The first sub-section presents requirements from the perspective of our users. This is followed by an overview of TAU. Finally, we layout implementation details of our plugin that allow us to *profile* parallel Java applications.

## Requirements of Profiler

In the context of MPJ Express users, the MPJProf tool must support the following requirements:

1.  *Basic features*: Provide basic profiling features such as time analysis of methods and statements

2.  *Instrumentation*: Must support both automatic and manual instrumentation

3.  *Scalable*: Must be scalable enough to allow profiling of large parallel programs executing on hundreds and thousands of processors.

4.  *MPJ Express modes*: Must support profiling of MPJ Express multicore and cluster modes.

5.  *Tracing*: Must be capable of generating event traces of parallel java applications.

With these goals in mind, a plausible approach to develop a profiler for MPJ Express is to exploit features of open-source performance analysis tool like TAU.

## Tuning and Analysis Utility (TAU)

Tuning and Analysis Utility (TAU) is a portable profiling and tracing toolkit for performance analysis of parallel programs written in Fortran, C, C++, UPC, Java, Python. TAU can profile parallel Java applications by collecting performance data of each method, statements and basic blocks for each thread, context, and node in use by an application. Using performance data, TAU can generate profiles for users which contain wealth of performance information. These profiles can give information about inclusive and exclusive time spent in each function in different time units, number of times function was called or how many functions were invoked by each function. Using this information user can easily identify performance bottlenecks in their applications.

Profiles generated by TAU, follow a special naming scheme `profile.<node>.<context>.<thread>`. TAU provides graphical and command line tools such as `paraprof` and `pprof` to visualize profiling results in nodes/threads and aggregated format. TAU can also generate event traces which follow the same name scheme `trace.<node>.<context>.<thread>`. Traces display when an event took place along a timeline. Typically users utilize open-source and free tools like `Vampir`, `Jumpshot`, and `Paraver` to visualize event traces. TAU also support some well-known parallel programming implementations like MPICH, Open MPI and mpiJava. As part of this paper, one of the objectives is to exploit TAU to profile Java applications parallelized using MPJ Express.

## MPJProf in Multicore Mode

Profiling of parallel Java applications running in multicore mode using TAU is straightforward. TAU uses `tau_java` to generate profiles for multithreaded Java applications. `tau_java` instruments Java applications at runtime using Java Virtual Machine Tool Interface (JVMTI). In multicore mode we have one node and multiple threads so generated profiles will be like; `profile.0.0.t` where `t` ranging from 0 to maximum number of threads being launched at the node. Use of `tau_java` is implemented in runtime system of MPJ Express software. If user has selected to profile parallel Java application then MPJ Express runtime system enable `tau_java` to start profiling.

## MPJProf in Cluster Mode

TAU can generate profiles for applications running in multicore mode by using `tau_java` but it cannot profile parallel Java applications running in cluster mode of MPJ Express. There is support for mpiJava in TAU to perform analysis of applications running at compute cluster. But mpiJava is implemented as a set of JNI wrappers to native MPI packages where MPJ Express is pure Java implementation of MPI. Profiles are not generated as per expectation with MPJ Express because value for "node" is hard-coded to zero in TAU source code and there is no option to change this value at runtime. Node represents process and in case of cluster mode, processes are distributed across nodes of cluster. So we need to change value of node according to processes. To achieve that we added **`-tau:node<NodeID>`** configuration option in `tau_java`. `NodeId` represents rank of MPJ process. When TAU runtime encounters this option it changes default value of node to option value. This setting of value is done in `tauJVMTI.cpp`

    if option is node {
        set tau profile node to option value
    }

This will change profile node to node id and profiles for that node will be generated. [^1]

In MPJ Express runtime if user has provided option to profile parallel Java application we start setting of profile node values by passing rank of each process to TAU runtime. As an example, four processes are distributed across two machines, so generatd profiles will be like `profiles.n.0.t`, where `t` ranges from 0 to total number of processes and `t` ranges from 0 to maximum threads launched by each node. Once profiles are generated, these can be viewed using graphical interface `paraprof` or command line interface `pprof`.

# Performance Analysis for NPB

MPJProf supports both profiling and tracing performance analysis for MPJ Epxress applications. Performance analysis results give user statistics of performance metrics and performance behavior. We performed analysis for Java NAS parallel benchmarks (NPB)   to provide useability of MPJProf. We used NPB kernel `IS` with workload `A` and run it under four processes. However, it should be understood MPJProf can be extended to larger number of processes.

In Figure  , we see various output windows of TAU's profile browser `paraprof`. For each profiling window metric is time and units are in seconds. We can see mean statistics of exclusive and inclusive time for all threads in Mean Data Statistics window sorted by exclusive time. Similarly we can view exclusive time for each node and thread in bar chart windows. Using this profiling data user can also see threads performing MPJ Express module and background JVM tasks which directly are not possible.

# Evaluation

In this section, we quantify the overhead of our profiling tool by employing variety of performance tests including basic latency and bandwidth benchmarks for point-to-point communication and Java NAS parallel benchmarks (NPB). We performed evaluation on following two test environments. The first test environment (from now on RCMS) consisted of a 32 node cluster hosted at RCMS-NUST, Pakistan. Each compute node contains two quad-core Intel Xeon E5520 processors with a main memory of 24G Bytes. The nodes are connected via Gigabit Ethernet. Its software environment consisted of the Oracle JDK 1.7.0_25 and TAU 2.23.1b. The second test environment (from now on 1GE) consisted of four machines, each having Intel^^ Corei5-3470 CPU with 3.20GHz and 8G Byte of memory. All four machines are connected through 1G Ethernet connection. Software environment of these machines consist of Oracle JDK 1.7.0.\_25 and TAU 2.23.1b. All systems are configured for optimized performance.

We performed standard latency and bandwidth test on RCMS. Right side of Figure   show throughput (bandwidth in Mbps) comparison across Gigabit Ethernet. MPJ Express achieves 83% of maximum bandwidth when executed without MPJProf. There is performance loss when bandwidth test is run using MPJProf. Maximum bandwidth achieved in that case is 81%. Left side of Figure   shows the latency (transfer time for one byte in ${\mu}$s) comparison across Gigabit Ethernet. The latency for MPJ Express without profiling is 57.3${\mu}$s and with profiling is 752${\mu}$s. The reason for higher latency with MPJProf is due to generation of profiling information by parallel processes.

We evaluated the performance of Java NAS parallel benchmarks (NPB) kernels on 1GE. We chose three NPB kernels namely `CG`, `IS` and `EP` and ran test on workload Class A on total of 16 processes using MPJ Express. Figure   shows performance analysis results time (in seconds) against NPB kernels using with and without MPJProf. We observed 15% of overhead added in kernel `CG`, 2% of overhead added in kernel `IS` and 1.5% of overhead added in kernel `EP`. Our results indicate that the MPJProf tool only adds a negligible overheads, which is due to generation of profiling information by parallel processes.

# Conclusions

MPJ Express is a Java messaging system that allows parallelizing applications on distributed memory platforms including compute clusters. In this paper we presented debugging and profiling tools named MPJDebug and MPJProf respectively. The main goal is to increase overall productivity of MPJ Express application developers. The first tool, MPJDebug, builds on the open-source Eclipse IDE that allows extending functionality of the core platform by building Eclipse plugins. MPJDebug has been developed as an Eclipse plugin and supports debugging parallel applications executing in multicore and cluster modes. The second tool, MPJProf, is a utility based on Tuning and Analysis Utility (TAU)—an open-source performance evaluation tool. MPJProf exploits TAU to profile Java applications parallelized using MPJ Express by generating profiles and traces, which can later be visualized using existing tools like `paraprof` and `Jumpshot`. We quantified the overhead of using MPJProf, which we found to be negligible in the profiling stage of parallel application development.

[^1]: This change was proposed to TAU developers. They approved it and added it in their beta release version 2.23.1b of TAU.