Title: Corral Framework: Trustworthy and Fully Functional Data Intensive Parallel Astronomical Pipelines,
Abstract: Data processing pipelines represent an important slice of the astronomical
software library that include chains of processes that transform raw data into
valuable information via data reduction and analysis. In this work we present
Corral, a Python framework for astronomical pipeline generation. Corral
features a Model-View-Controller design pattern on top of an SQL Relational
Database capable of handling: custom data models; processing stages; and
communication alerts, and also provides automatic quality and structural
metrics based on unit testing. The Model-View-Controller provides concept
separation between the user logic and the data models, delivering at the same
time multi-processing and distributed computing capabilities. Corral represents
an improvement over commonly found data processing pipelines in Astronomy since
the design pattern eases the programmer from dealing with processing flow and
parallelization issues, allowing them to focus on the specific algorithms
needed for the successive data transformations and at the same time provides a
broad measure of quality over the created pipeline. Corral and working examples
of pipelines that use it are available to the community at
this https URL.