Title: Exploiting Apache Spark platform for CMS computing analytics,
Abstract: The CERN IT provides a set of Hadoop clusters featuring more than 5 PBytes of
raw storage with different open-source, user-level tools available for
analytical purposes. The CMS experiment started collecting a large set of
computing meta-data, e.g. dataset, file access logs, since 2015. These records
represent a valuable, yet scarcely investigated, set of information that needs
to be cleaned, categorized and analyzed. CMS can use this information to
discover useful patterns and enhance the overall efficiency of the distributed
data, improve CPU and site utilization as well as tasks completion time. Here
we present evaluation of Apache Spark platform for CMS needs. We discuss two
main use-cases CMS analytics and ML studies where efficient process billions of
records stored on HDFS plays an important role. We demonstrate that both Scala
and Python (PySpark) APIs can be successfully used to execute extremely I/O
intensive queries and provide valuable data insight from collected meta-data.