Title: Enabling near real-time remote search for fast transient events with lossy data compression,
Abstract: We present a systematic evaluation of JPEG2000 (ISO/IEC 15444) as a transport
data format to enable rapid remote searches for fast transient events as part
of the Deeper Wider Faster program (DWF). DWF uses ~20 telescopes from radio to
gamma-rays to perform simultaneous and rapid-response follow-up searches for
fast transient events on millisecond-to-hours timescales. DWF search demands
have a set of constraints that is becoming common amongst large collaborations.
Here, we focus on the rapid optical data component of DWF led by the Dark
Energy Camera (DECam) at CTIO. Each DECam image has 70 total CCDs saved as a
~1.2 gigabyte FITS file. Near real-time data processing and fast transient
candidate identifications -- in minutes for rapid follow-up triggers on other
telescopes -- requires computational power exceeding what is currently
available on-site at CTIO. In this context, data files need to be transmitted
rapidly to a foreign location for supercomputing post-processing, source
finding, visualization and analysis. This step in the search process poses a
major bottleneck, and reducing the data size helps accommodate faster data
transmission. To maximise our gain in transfer time and still achieve our
science goals, we opt for lossy data compression -- keeping in mind that raw
data is archived and can be evaluated at a later time. We evaluate how lossy
JPEG2000 compression affects the process of finding transients, and find only a
negligible effect for compression ratios up to ~25:1. We also find a linear
relation between compression ratio and the mean estimated data transmission
speed-up factor. Adding highly customized compression and decompression steps
to the science pipeline considerably reduces the transmission time --
validating its introduction to the DWF science pipeline and enabling science
that was otherwise too difficult with current technology.