Title: An Automated Scalable Framework for Distributing Radio Astronomy Processing Across Clusters and Clouds,
Abstract: The Low Frequency Array (LOFAR) radio telescope is an international aperture
synthesis radio telescope used to study the Universe at low frequencies. One of
the goals of the LOFAR telescope is to conduct deep wide-field surveys. Here we
will discuss a framework for the processing of the LOFAR Two Meter Sky Survey
(LoTSS). This survey will produce close to 50 PB of data within five years.
These data rates require processing at locations with high-speed access to the
archived data. To complete the LoTSS project, the processing software needs to
be made portable and moved to clusters with a high bandwidth connection to the
data archive. This work presents a framework that makes the LOFAR software
portable, and is used to scale out LOFAR data reduction. Previous work was
successful in preprocessing LOFAR data on a cluster of isolated nodes. This
framework builds upon it and and is currently operational. It is designed to be
portable, scalable, automated and general. This paper describes its design and
high level operation and the initial results processing LoTSS data.