Scalability Model for the LOFAR Direction Independent Pipeline
A. P. Mechev T. W. Shimwell A. Plaat H. Intema A. L. Varbanescu H. J. A Rottgering









apmechev@strw.leidenuniv.nl








 
^a Leiden Observatory, Niels Bohrweg 2, 2333 CA Leiden, the Netherlands
^b ASTRON, Oude Hoogeveensedijk 4, 7991 PD , The Netherlands 
^c Leiden Institute of Advanced Computer Science, Niels Bohrweg 1, 2333 CA Leiden, the Netherlands
^d International Centre for Radio Astronomy Research – Curtin University, GPO Box U1987, Perth, WA 6845, Australia
^e University of Amsterdam, Spui 21, 1012 WX Amsterdam, the Netherlands



LOFAR is a leading aperture synthesis telescope operated in the Netherlands with stations across Europe. The LOFAR Two-meter Sky Survey (LoTSS) will produce more than 3000 14 TB data sets, mapping the entire northern sky at low frequencies. The data produced by this survey is important for understanding the formation and evolution of galaxies, supermassive black holes and other astronomical phenomena. All of the LoTSS data needs to be processed by the LOFAR Direction Independent (DI) pipeline, . Understanding the performance of this pipeline is important when trying to optimize the throughput for  large projects, such as LoTSS and other deep surveys. Making a model of its completion time will enable us to predict the time taken to process large data sets, optimize our parameter choices, help schedule other LOFAR processing services, and predict processing time for future large radio telescopes. We tested the  pipeline by scaling several parameters, notably number of CPUs, data size and size of calibration sky model. We present these results as a comprehensive model which will be used to predict processing time for a wide range of processing parameters. We also discover that smaller calibration models lead to significantly faster calibration times, while the calibration results do not significantly degrade in quality. Finally, we validate the model and compare predictions with production runs from the past six months, quantifying the performance penalties incurred by processing on a shared cluster. We conclude by noting the utility of the results and model for the LoTSS Survey, LOFAR as a whole and for other telescopes. 






Radio Astronomy Performance Analysis Performance Modelling High Performance Computing Scalability















§ INTRODUCTION 



Astronomy has entered the big data era with many projects creating petabytes of data per year. This data is often processed by complex multi-step pipelines consisting of various algorithms. Understanding the scalability of astronomical algorithms theoretically, in a controlled environment, and in production is important for making predictions for the data reduction of future projects and upcoming telescopes. 

The Low Frequency Array (LOFAR) <cit.> is a leading European low-frequency radio telescope. The majority of LOFAR's stations are in the Netherlands, however the telescope can use stations across Europe to create ultra-high resolution radio maps. LOFAR data needs to undergo several computationally intensive processing steps before obtaining a final scientific image. 

To create a broadband image, LOFAR data is first processed by a Direction Independent (DI) Calibration pipeline followed by Direction Dependent (DD) Calibration pipeline <cit.>. The goal of DI calibration is to remove effects that are constant across the target field such as radio frequency interference, contamination by bright off-axis sources and antenna gains. After this step, DD Calibration focuses on removing effects which vary across the field, such as ionospheric and beam effects. The result of these two pipelines is a science-ready image. 

Our implementation of the DI LOFAR processing, , can be parallelized on a high throughput cluster <cit.>. The Direction Dependent processing, implemented in [Available at https://github.com/mhardcastle/ddf-pipeline/releaseshttps://github.com/mhardcastle/ddf-pipeline/releases], is subsequently performed on a single HPC node. 

The LOFAR Surveys Key Science Project (SKSP) <cit.> is a long running project consisting of several low frequency surveys of the northern sky. The broadest tier of the survey, LoTSS, will use more than 3000 8-hour observations to create maps with a noise levels below 100 μJy. We have already processed more than 500 of these observations using the  DI pipeline <cit.>. 

While the current LoTSS imaging algorithms can process data averaged by up to a factor of 64 in frequency and time, it is important to understand how LOFAR processing scales with processing parameters, such as averaging parameters. Since LOFAR data is used by multiple scientific teams, not every team can produce scientific results from data averaged by such a high factor. Users from those teams need to be able to predict the time and computational resources required to process their data, taking into account the increasing LOFAR observation rates, data sizes and scientific requirements. 

We study the scalability of processing LOFAR data, by setting up processing of a sample SKSP data set on an isolated node on the  cluster at SURFsara, part of the Dutch national e-infrastructure <cit.>. We test the software performance as a function of several parameters, including averaging parameters, number of CPUs and calibration model size. Additionally, we test the performance of the underlying infrastructure, i.e. queuing  and download time, for the same parameters. Finally, we compare those isolated tests with our production runs of the  pipeline to measure the overhead incurred by running on a shared system. 

We discover that the computationally intensive LOFAR processing steps scale linearly with data size, and calibration model size. Additionally, we find that the time taken by these steps is inversely proportional to the number of CPUs used. We discover that the time to download and extract data on the  cluster is linear with size up to 32GB, but becomes slower beyond this data size. We also find that the queuing time on the  cluster grows exponentially for jobs requesting more than 8 CPUs. We validate these isolated tests with production runs of LOFAR data from the past six months. We combine all these tests into a single model and show its prediction power by testing the processing time for different combinations of parameters. Finally, we discuss the utility of our method, the results in this work and applications to the SKSP projects, the broader impact of our results to LOFAR processing and the applications for other large astronomical surveys. The major contributions of this work can be summarized as:


    
  * A model of processing time for the  LOFAR Direction Independent Calibration Pipeline.
    
  * A model of queueing time and file transfer time which is used by current or future jobs processed on the  cluster.
    
  * Quantification of overheads incurred when processing in production. 
    
  * Validation of our methods with discussion of future applications. 


We introduce LOFAR processing and other related work in Section <ref> and describe our software setup and data processing methods in Section <ref>. We present our results and performance model in Section <ref> and discussions and conclusions in Section <ref>. 





§ RELATED WORK

In previous work, we have parallelized the Direction Independent LOFAR pipeline on a High Throughput infrastructure <cit.>. While this parallelization has helped accelerate data processing for the SKSP project, creating a performance model of our software is required if we are to predict the resources taken by future jobs. This model will be particularly useful in understanding how processing parameters will affect run time.  

In previous work, we have parallelized the Direction Independent LOFAR pipeline on a High Throughput infrastructure <cit.>. While this parallelization has helped accelerate data processing for the SKSP project, creating a performance model of our software is required if we are to predict the resources taken by future jobs. This model will be particularly useful in understanding how processing parameters will affect run time.  

Performance modelling on a distributed system is an important field of study related to grid computing. A good model of the performance of tasks in distributed workflows can help more efficiently schedule these jobs on a grid environment <cit.>. The performance modeling systems require knowledge of the source code and an analytical model of the slowest parts of the code <cit.>. Many systems exist to model the performance of distributed jobs <cit.>, with some employing Black Box testing <cit.> or tests on scientific benchmark cases <cit.>. Such performance analysis does not require intimate knowledge of the software and can be applied on data obtained from processing on a grid infrastructure.

Empirical modelling is useful in finding performance bugs in parallel code <cit.> and modelling the performance of big data architectures <cit.>. The insights from these models are used to optimize the architecture of the software system or determine bottlenecks in processing. Here, we use empirical modelling to determine how the LOFAR  performance scales with different parameters. 



§ PROCESSING SETUP 


=10000
Using the LOFAR software installation described in <cit.>, we processed a typical LOFAR SKSP observation[LOFAR Observation ID L658492, co-ordinates [17h42m21.785, +037d41m46.805] observed by the LOFAR High Band Array for 8 hours between 2018-06-20 and 2018-06-21.], while changing the averaging rate  in time and frequency. Changing these averaging parameters will change the final data size (with the data sizes studied shown in Table <ref>). We test the processing time for different averaging parameters by running 15 runs per parameter step. 

The data used by the LOFAR surveys is archived at a time resolution of 1 second intervals and frequency resolution of 16 channels per subband (equivalent to 12kHz channel width). While some of the processing steps such as flagging of Radio Frequency Interference and removal of bright off-axis sources produce better results when performed on the high-resolution data, later steps can be performed on averaged data with little impact on the final product quality. To speed up processing, the raw data is averaged in time and frequency, decreasing the input data size to later tasks. The main aims of the LoTSS survey can be accomplished if the final data products from the  pipeline are averaged to a time resolution of 8 seconds per sample and frequency of 2 channels per subband. These averaging parameters correspond to a reduction in data size by a factor of 64. Nevertheless, other science cases require less averaging of the data. Our aim is to understand how the processing of this larger data will scale. To measure the scalability of processing, we measure the performance of the  pipeline for data sizes between the raw data of 64GB/subband and the averaged data of 1GB/subband. The tested data sizes and parameters are shown in table <ref>, and discussed in Section <ref>.

We performed the scalability tests on a dedicated node of the SURFsara  cluster, f18-01. The node is a typical hardware node used by our production LoTSS processing, however it is dedicated for the tests in order to ensure there is no contamination by other software. The node is described in Section <ref>. 
     



We processed the sample data set with the LOFAR  pipeline. The  version used was the same as we use for the LOFAR SKSP broadband surveys <cit.>. This software consists of several steps executed in sequence, shown graphically in Figure <ref>. The important  steps are as follows. The qcrpredict_ateam and qcrateamcliptar steps predict the contamination by bright off-axis sources and remove these effects respectively. The qcrdpppconcat step is responsible for concatenating 10 subbands into a single file which is in turn calibrated. The step qcrgsmcal_solve is responsible for calibration of the data against a model of the radio sky. The solutions produced by qcrgsmcal_solve are used by qcrgsmcal_apply and applied to the scientific observation.





 §.§ Processing Metrics

The goal for our scalability model is to understand the effect of several parameters on the job completion time of LOFAR software. We do this by testing the processing time for various values of data size, number of CPUs used and sky model size. 
 



The slowest step of the  pipeline is the qcrgsmcal_solve step, which performs the gain calibration against a model of the radio sky. This step operates on a concatenated data set that consists of 10 subbands. We obtain the calibration model through the TGSS sky model creator[Accessible at http://tgssadr.strw.leidenuniv.nl/doku.phpthe TGSS ADR portal.]. By default this service creates a text file describing the sky-model from the TGSS survey <cit.> as a combination of gaussian and point sources. By default, it sets a threshold of sources brighter than 0.3 Jy. 
Lowering this threshold creates longer sky-model files with more faint sources, while increasing it will return only the few brightest sources. Since sky model calibration requires converting the sky-model into modelled visibilities <cit.>, a longer sky model will increase the time taken to gain calibrate a data set. We created 7 sky models with a flux cutoff ranging between 0.05 Jy and 1.5 Jy. The number of sources in the resulting models are listed in Table <ref>. 
For production[The query used to obtain model 3 is at the following link <http://bit.ly/tgss_model>], we used the minimum sensitivity parameters for model 3.


It is important to note that the complexity and accuracy of the sky model depend on the direction of observation and the ionospheric conditions in which the observation was performed. As such, our test is only a heuristic for predicting the run-time based on the calibration model length. Additionally, it is notable that the number of sources is exponentially dependent on the minimum sky model sensitivity (seen in Figure <ref>, more in ). According to this relationship, even a modest decrease in sensitivity cutoff can significantly decrease the size of the model.







Finally, the number of CPU cores (henceforth just `CPUs') used by each step is a parameter that can be optimized for the entire pipeline. While increasing the number of CPUs can make some steps run faster, requesting jobs that reserve a large number of CPUs will take longer to launch on shared infrastructure. In order to understand the interplay between these effects, we study the queuing time and processing time as a function of the number of CPUs. For the parameter steps we choose to test 1, 2, 3, 4, 8 and 16 CPUs. 



 §.§ Infrastructure Performance



In production, we run hundreds of LoTSS jobs on a cluster supporting several different use cases. The requested resources on this cluster are allocated by a job queue, in our case implemented by the glite workload management system <cit.>. As queuing jobs can take a significant amount of time, we test the queuing time as a function of the number of requested CPUs. In order to do that, we create test jobs that log the launch time and submit them, requesting 1, 2, 3, 4, 8 and 16 CPUs. We run 10 to 15 tests for each parameter step to ensure that we capture system variability at different times of day during the week and the weekend. 

Besides queuing, time is also spent during downloading and unpacking data, as well as packing and uploading the results. Despite using no compression to pack the data, untarring and tarring large files still takes time depending on the system workload. We measure the time taken to transfer and unpack data of different sizes. The data sizes we chose were 0.5GB, 1GB, 2GB, 4GB, 8GB, 16GB, 32GB and 64GB. As our largest data sets are 64GB and our smallest results are ∼0.2GB, these values span a realistic range expected for LOFAR data processing. We test this by uploading mock data to the dCache storage pool at SURFsara and launching a small 1 CPU job on the cluster, which downloads and untars the data and logs the start time of each step. We present the results of this test in the next section. 




  §.§.§ Software Versions

For the current test, we use the LOFAR software stack, version 2.20.2 <cit.>. This software was compiled on a virtual machine and distributed using the CERN CVMFS virtually mounted file system <cit.>. We use this software version and distribution method as it is the same software version and distribution used to process the data for the LOTSS Data Release 1. 



 §.§ Test Hardware


We test the LOFAR software on a reserved node on the SURFsara  cluster. The node, f18-01 has 348 GB of RAM, 3TB of scratch space[More detailed specifications are at http://docs.surfsaralabs.nl/projects/grid/en/latest/Pages/Service/system_specifications/gina_specs.htmlthe  specification page linked here]. The CPU is an Intel Xeon(R) Gold 6148 CPU with 40 cores clocked at 2.40GHz.  As this hardware node was reserved, there was no other scientific processing aside from our tests, meaning there was no resource contention aside for that inherent in the LOFAR software. In the results section, we compare these isolated runs with processing results over the past two years. 



§ RESULTS



Using a test data set, we tested the LOFAR  target pipeline on the SURFsara  cluster. First we will present the tests done in an isolated environment and then compare them to the run time in production on a shared infrastructure. We will integrate all the results in a complete model which can be used to predict processing time for a variety of parameters. Finally, we will make some predictions on the run time of our processing based on the model and validate these predictions. 

Since we are  processing a sample data set in the context of the LOFAR Surveys project, we will compare these tests with the production runs of our pipeline. In production, we run  the qcrgsmcal_solve step with a data size of 1GB, a sky model with 180 sources and 8 CPUs. 
 



 §.§ Isolated Environment tests

We first tested the LOFAR software in isolation in order to determine the scalability of processing time in terms of data size. We run the entire  target pipeline which which removes Direction Independent Calibration errors from a LOFAR science target. In the following sections, we present the models obtained from these tests.  



  §.§.§ Input Data Size

LOFAR data can be averaged to different sizes based on the scientific requirements. Smaller data sets are processed faster, so it is important to understand the effect of data size on processing time as measured by wall-clock time. We show the processing time for our test data set, averaged to different sizes for several  steps in Figures <ref>- <ref> and <ref>. We run this test using 8 CPUs. The figures also show linear fits for consecutive pairs of parameter steps, in gray dashed lines, used to help guide the selection of parametric model. 

All of the steps show a linear behavior with respect to input data size, while the qcrgsmcal_solve step is best fit by two linear relationships, for data smaller and larger than 16 GB. The linear fit to the run times are shown in Equations <ref>-<ref>. The equations show the processing time as a function of the data size (𝒮), with the slope in the units of seconds/byte. The fits are also shown in Figures <ref> to <ref> as a black dashed line.




    T_predict_ateam=5.19×10^-8𝒮+4.20×10^1 
    
            T_ateamcliptar=4.57×10^-9𝒮-8.42×10^0 
    
            T_dpppconcat=3.51×10^-8𝒮+4.20×10^1 
    
            T_gsmcal_solve=
                              7.38×10^-7𝒮-8.20×10^1    |𝒮≤1.6×10^10
    
                              1.04×10^-6𝒮-4.04×10^3     |𝒮>1.6×10^10
    
            T_gsmcal_apply=2.07×10^-8𝒮-1.38×10^1



Equations describing the processing time of five  steps as a function of the input data size (𝒮) in bytes.










  §.§.§ Calibration Model Size

To test the effect of the calibration model size on run time, we test our calibration with several different lengths of the sky model file. We create these models by changing the minimum sensitivity using values ranging from 0.05 Jy to 1.5 Jy. The most sensitive model (0.05 Jy) had 809 sources while the 1.5 Jy model had only 16 sources. 

Figure <ref> shows that the calibration time is directly proportional to the length of the sky model. Figure <ref> shows the run time as a function of the processing parameter: the cutoff sensitivity. As the relationship between the number of sources and cutoff sensitivity is a power law, here we see the same relationship holding for processing time.

We model the processing time as a function of the cutoff frequency using a power law, and fit the data to the function y=α·ℱ^-k. Our fit found the best model to be shown in Equation <ref>, where ℱ value is the cutoff flux in Jansky and T is the run time in seconds. 

We show four images made from data sets in Figure <ref>. The top left image is calibrated with a 0.05Jy and the other three show the difference between the top left image and the images created from the 0.3Jy, 0.8 Jy and 1.5 Jy data. The statistics for the four images, taken from the regions in green on Figure <ref>) are shown in Table <ref>. We discuss the implication and caveats of these results in Section <ref>.






    T=1185·ℱ^-0.854

Processing time for the qcrgsmcal_solve step as a function of the flux cutoff of the calibration model (ℱ) in Jansky







Using the command wsclean -absmem 50 -niter 3 -size 4096 4096 -scale 20asec





  §.§.§ Number of CPUs

One further parameter that can be optimized is the number of CPUs requested when the job is launched. We investigated the processing speedup as a function of the number of CPUs for the  target pipeline. From the steps tested, only the qcrgsmcal_solve step shows a significant speedup as the number of CPUs is increased. The run time of this step is an inverse power law with respect to the number of CPUs as seen in Figure <ref>. Unlike the solving step, the step applying the calibration solutions (qcrgsmcal_apply) is constant in time with respect to the number of CPUs as seen in Figure <ref>. The difference in performance is most likely because the qcrgsmcal_apply code uses a parallel for loop to calculate antenna gains while qcrgsmcal_apply does not. 

We fit a model with processing time inversely proportional to the number of CPUs used. We show the resulting model in Equation <ref>, with the parameter (𝒩) being the number of CPUs used. 







    T=503.37+3062.6/𝒩

Processing time for the qcrgsmcal_solve step as a function of (𝒩), the Number of CPUs used by the process.





 §.§ Queuing Tests


Aside from performance of the LOFAR software, we measured the queuing time at the  cluster, as a function of the number of CPUs requested. This data was obtained between 16 Nov 2018 and 10 Dec 2018 for 1,  2, 3 ,4, 8, and 16 CPUs per job. A histogram of the queuing time for these jobs is shown in Figure <ref>. Statistics for these runs are in Table <ref>. We use the 75th percentile of the queuing time for each parameter step to fit a model. This scenario will include 75% of runs and is a good trade-off between ignoring and including outliers. 

We fit two linear models for this queueing time. One model for 1-4 CPUs and one for 4-16 CPUs. The model, as a function of the number of CPUs 𝒩 is in equation <ref>. The two models are plotted against the 75th percentile of the queuing times (last column in Table <ref>) in Figure <ref>.










    T = 
        49.3·𝒩+ 120    |𝒩≤4
    
        726·𝒩-3071     |𝒩>4

The model for the Queuing time as described by two linear models. 






 §.§ Transfer and Unpacking Time


We tested the downloading and unpacking time for data sizes ranging from 512MB to 64GB. We discovered that the unpacking of files below 64GB scaled linearly with file size, however unpacking individual data sets larger than 16GB becomes considerably slower than downloading it. 

Figure <ref> shows the histogram of the download tests, and Figure <ref> displays the tests as a function of data size. Both figures show that extracting of the 32 and 64GB data sets has more slow outliers than the downloading of this data. 

We fit a power law model to the time taken to transfer and unpack the data. In this case, we also consider the 75th percentile of these times in order to capture the majority of runs and ignore outliers. The plot of the data and our model can be seen in Figure <ref> and the model is in Equation <ref>, as a function of the input data size,
𝒮 in gigabytes.









    T=5.918×10^20·𝒮^2.336

Model of the downloading and extracting time as a function of the data size (𝒮) in bytes.





 §.§ Comparison with production runs

Over the past two years, the LOFAR software has been running in production and collecting data on run time for each processing step. We have saved detailed logs for these runs starting in July 2018.  We can compare this to the isolated model in order to determine the overhead incurred by processing LOFAR data on shared nodes. 

Using the logs recorded by our processing launcher [GRID_PiCaS_Launcher, <https://github.com/apmechev/GRID_picastools>], we made plots showing the processing time for the downloading and extracting, and for the slowest steps, qcrndppp_prepcal and qcrgsmcal_solve. 
The results are shown in Figures <ref> and <ref>. We include predicted extract times from Section <ref> as vertical dashed lines for both plots. The agreement between our model and production runs are an encouraging result for future software performance modelling. 

Finally, we present Figure <ref> which shows a comparison of qcrgsmcal_solve run times and our model's prediction for a 1GB data set.  Figure <ref> plots the processing time vs data size for these production runs and includes the model from Equation <ref>. The significant overhead incurred on a shared infrastructure can be noted. 















 §.§ Complete Scalability Model

To incorporate all our data into a complete model, we consider the slowdown of each parameter as a multiplier to the time taken to process our base run. We incorporate the models for each parameter above for the model of the run time. We add the transfer and queuing time to the processing time to obtain a final function of all our parameters. We can use this function to predict the processing time for an arbitrary data set. 

The final performance model for the slowest steps, qcrgsmcal_solve, qcrdpppconcat  and qcrpredict_ateam are in Equation <ref>. 

[!t]



    t_infrastructure    = 
            49.3·𝒩+ 120    |𝒩≤4
    
            726·𝒩-3071     |𝒩>4
           
        + 2 · 0.056·𝒮^2.336

     

    t_gsmcal_solve    = t_infrastructure
        + [3566·1/3.012ℱ^-0.854· (0.1412+0.8589/𝒩) ]  · 
                 7.38·10^-7𝒮 | 𝒮≤16
    
                 1.04·10^-6𝒮 | 𝒮>16

 
 
    t_dpppconcat   = t_infrastructure
        + 3.51×10^-8𝒮+4.20×10^1

 
  
    t_predict_ateam   = t_infrastructure
        + 5.19×10^-8𝒮+4.20×10^1



Model of the total time of the most computationally expensive steps for the parameters 𝒩, Number of CPUs; 𝒮, Size of data in bytes and ℱ, cutoff calibration model flux in Jansky. These models include processing times, as well as infrastructure overheads. As the model for the queuing, downloading and uploading time does not change for different processing steps, we decide to keep it separate for clarity. The complete scalability models for the rest of the steps can be derived similarly, however are omitted here as they consist of the minority of processing time for LOFAR DI processing.  





§ DISCUSSIONS AND CONCLUSIONS



The goal of this work is to understand the performance of the LOFAR Direction Independent Pipeline as processing parameters are changed. We modify several parameters and compare the wall clock time taken to process the data. Finally, we study data from queuing jobs and downloading data, in order to fully model infrastructure overheads. We compare our model with past runs of the software and discuss the results and implications. We discuss the utility of this model for current and upcoming LOFAR projects. Finally, we note the effectiveness of this modelling technique for understanding the performance of large-scale processing of astronomical data.



 §.§ Software Performance


We performed several tests to determine the scalability of LOFAR processing with respect to several parameters. We outline our findings below as well as their implications for processing in context of the LOFAR surveys and other LOFAR projects. 



  §.§.§ Data Size

We tested broadband LOFAR data ranging in size by a factor of 64, and discover that all our processing steps scale linearly in time with respect of the input data size. We learn that for input data above 16GB, the slope of our scaling relation is higher than for the smaller data sets. The linear scaling of our processing suggests that projects interested in processing massive LOFAR data sets can scale well in terms of processing time. 

As the calibration step concatenates 10 input subbands, data larger than 16GB shows a higher slope (Figure <ref>), meaning they take longer to process than smaller data. We compared the 10GB run and the 64GB run, which showed that the latter fills the system's page cache, while the former does not. It is likely that this memory usage is related to the overhead seen in Figure <ref> and 16GB break seen in Figure <ref>. Including this break in the model also helps make a more accurate processing time prediction as fitting a single linear model would have a large negative y-intercept, predicting negative processing times for data smaller than 2GB. 

Our analysis shows the following results. Overall, the slowest step was the qcrgsmcal_solve step, and its run time scales more strongly with data size than the other steps (equation <ref> has the steepest slope). This suggests that as data sizes increase, qcrgsmcal_solve will increasingly dominate the processing time over the other steps (As seen in Figures <ref> and <ref>). This effect is especially prominent for data larger than 16GB (160GB when 10 subbands are concatenated). As such, it is recommended to avoid calibration of data larger than 160GB. This limitation suggests that science requiring operations on non-averaged data sets, such as long-baseline imaging, will require significant computational requirements for high-fidelity images.



  §.§.§ Calibration Model Size

We discover that the calibration time scales linearly in as a function of the length of the calibration model, however as a power law with respect to the model's cutoff sensitivity. This is because of the (expected) power law relation between the number of sources and cutoff sensitivity, seen in Figure <ref>. We can use this discovery to accelerate the processing time by increasing the flux cutoff to the LOFAR direction independent calibration to 0.5 Jy. Doing so will execute the calibration step in 60% of the time, saving 83 CPU-h per run. Over the remaining 2000  runs left in the LOTSS project, this change in sensitivity will save more than 167k CPU-hours. 

Figures <ref> show a data set calibrated with sky models with cutoff sensitivities listed in Table <ref>, and figures  <ref> and <ref> show the calibration solutions obtained by calibrating with skymodels of cutoff ranging from 0.05 Jy to 1.5 Jy. These results suggest that performing gain calibration with less complex, and thus smaller, calibration models will not degrade image and solution quality while taking less than 20% of processing time. Table <ref> also confirms this result. 

While this result is encouraging, there are caveats suggesting future study is required. The results we present are for a single observation, and has not been processed through the Direction Dependent Calibration pipeline. This pipeline produces final images used for scientific research. Future work will need to confirm that smaller calibration models used in the  pipeline do not degrade the quality of these final images. Nevertheless if this result holds, the calibration model threshold can be decreased for a wide range of LOFAR projects, significantly saving processing time and computing resources. 



  §.§.§ Comparison with production runs


When comparing our model's prediction with real processing runs over the past six months, we note that there are considerable overheads when running on a shared infrastructure vs. when processing data on an isolated (Figure <ref>). The overhead in processing is roughly a factor of two-three from our model. This discrepancy suggests that a model for qcrgsmcal_solve needs to be built using data when running on a shared environment, to better predict processing time. 



 §.§ Infrastructure Performance

We tested downloading and extracting LOFAR data of various sizes. Both downloading and extracting are shown to be linear in time with respect to the data size for data up to 32 GB. Beyond those sizes, there is more scatter in data extraction due to high file-system load. This is because load on the worker node's file-system can be unpredictable and can affect the data extraction times negatively. Nevertheless, when comparing our extraction tests and processing for the past 6 months, the predictions by our models (Figure <ref>) correspond to the production runs (Figures <ref> and <ref>).  


Part of the LOFAR SKSP processing is done on shared infrastructure, which requires requesting processing time ahead of time for each grant period. Being able to predict the amount of resources required to process data each grant period is required to make a reliable estimate on what resources to request. These results can be used by other projects sharing the SURFsara  cluster to predict their processing time before submitting jobs. 



§ APPLICATIONS AND CONCLUSIONS


Our performance model shows that it is possible to predict the processing time and computational resources used by a complex astronomical pipeline. Our results suggest that LOFAR LoTSS processing can be further optimized without sacrificing quality of the final product. Additionally, our results are transferable to other scientific pipelines that process LOFAR data with different parameters.  Any pipeline that performs gain calibration or application of calibration solutions will benefit from these results. 

In order to provide LOFAR processing as a service to scientific users, we need to estimate the processing time for each request. We need this estimation in order to determine whether the user has sufficient resources left in their resource pool. Knowing the performance of the software pipelines as a function of the input parameters will help predict the run time for each request and the resources consumed. Knowing this will make it possible to notify users how long the request will take and how much of their quota will be depleted. It is important to note that while this model is specific to LOFAR processing, the method we detail can be used by other scientific teams in order to predict the computational requirements for their pipelines. Doing this is necessary if large scale scientific processing is to be offered as a service to the wider community.

Finally, a performance model of the LOFAR software will help make predictions on the time and resources needed to process data for other telescopes such as the Square Kilometer Array (SKA). Once operational, the SKA is expected to produce Exabytes of data per year. Processing this data efficiently requires understanding the scalability of the software performance to facilitate scheduling and resource allotment. Overall, we show that our method helps guide algorithm development in radio astronomy, can be used to predict resource usage by complex pipelines and will be promising in optimizing data processing by future telescopes. 









§ CALIBRATION SOLUTIONS FOR THE SKY MODEL TESTS 

The output of the calibration step is a data set corrected for direction independent effects, as well as a set of calibration solutions. Figures <ref> and <ref> show the calibration solutions for core stations obtained when calibrating with sky models with minimum flux cutoffs of 0.05, 0.3, 0.8 and 1.5Jy. Much like in Figure <ref>, we can see that there is no significant difference between the calibration solutions for these stations. As a note, the naming scheme for LOFAR stations is CS/RS for core/remote stations, three digits for station number, HBA0/HBA1 for High Band antennas and LBA0/LBA1 for low-band antennas. The 0,1 suffixes correspond to sub-arrays in the core stations which can be correlated separately. Additionally, the CS/RS is replaced with the 2-letter country code for international stations<cit.>.

We compare the phase solutions for stations CS032HBA0 and CS003HBA0 and the reference station for two different calibrations in Figure <ref>. We note that this difference is within 0.3 rad for the entire observation. Combined with the results in the other two plots, our results suggest that the calibration solutions do not degrade when calibration is done with a smaller sky model.









§ PARAMETRIC MODEL PARAMETERS AND FIT ACCURACY


In this section, we note the uncertainties to the models fit in Equations <ref>-<ref>. 


equationsection





 §.§ Fits quality of run time vs input size model


The models of the processing time vs input size were fit as a linear regression. In this work we present such models for the qcrgsmcal_solve, qcrgsmcal_apply, qcrdpppconcat, qcrpredict_ateam and qcrateamcliptar, the five slowest steps. The resulting models, calculated by the <cit.> routine, are shown in Equation <ref>. We present the R^2 values, P values and standard error below, in Table <ref>.








 §.§ Fit of run time vs calibration model flux cutoff 


The run time vs Flux cutoff model shown in Equation <ref> is defined by the equation y=a· x^-k and two parameters, a and k. The covariance matrix for these two parameters is shown in Equation <ref>. The standard deviation for the fit of the parameters a and k is 26.134 and 7.624×10^-3 respectively.



    [   6.83×10^2 -1.94×10^-1; -1.94×10^-1  5.81×10^-5;             ]

The covariance matrix of the parameters in model in Equation <ref>.





 §.§ Fit of the NCPU model 

The covariance matrix for the fit parameters of equation <ref>, a and k in y=a+k/𝒩 are shown in Equation <ref>. The standard deviation of the fits for a and k are 13.11 and 48.20 respectively. 



    [  171.94 -504.11; -504.11 2322.95 ]

The covariance matrix for the parameters for the model predicting run time vs Number of CPUs used, shown in Equation <ref>.





 §.§ Fit for the queuing time model


The statistics of the model fit parameters for the queuing time model (Equation <ref>) are in Table <ref>. 
The queuing model is fit to the 75^th percentile of the queuing times for each parameter step. Since this results in a single number for each step, the model's P values are larger than the models from the other sections. 






 §.§ Fit of the download and extract model 

Equation <ref> shows the covariance matrix for the two parameters  a and k, y=a×10^k with the best fit values shown in Equation <ref>. The standard deviations of the fits for a and k are 0.016 and 0.068 respectively. 



    [ 2.53×10^-4 1.08×10^-3; 1.08×10^-3 4.60×10^-3 ]

The covariance matrix for the parameters for the model for Download and Extract time, shown in Equation <ref>.






§ ACKNOWLEDGMENTS

APM would like to acknowledge the support from the NWO/DOME/IBM programme “Big Bang Big Data: Innovating ICT as a Driver For Astronomy”, project #628.002.001.

This work was carried out on the Dutch national e-infrastructure with the support of SURF
Cooperative through grants e-infra 160152 and e-infra 180169.



§ REFERENCES



