Two-phase I O is a well-known strategy for implementing collective MPI-IO functions. It redistributes I O requests among the calling processes into a form that minimizes the file access costs. As modern parallel computers continue to grow into the exascale era, the communication cost of such request redistribution can quickly overwhelm collective I O performance. This effect has been observed from parallel jobs that run on multiple compute nodes with a high count of MPI processes on each node. To reduce the communication cost, we present a new design for collective I O by adding an extra communication layer that performs request aggregation among processes within the same compute nodes. This approach can significantly reduce inter-node communication congestion when redistributing the I O requests. We evaluate the performance and compare with the original two-phase I O on a Cray XC40 parallel computer with Intel KNL processors. Using I O patterns from two large-scale production applications and an I O benchmark, we show the performance improvement of up to 29 times when running 16384 MPI processes on 256 compute nodes.
@cite_26: 
@cite_21: Client-side file caching has long been recognized as a file system enhancement to reduce the amount of data transfer between application processes and I O servers. However, caching also introduces cache coherence problems when a file is simultaneously accessed by multiple processes. Existing coherence controls tend to treat the client processes independently and ignore the aggregate I O access pattern. This causes a serious performance degradation for parallel I O applications. In this paper we discuss our new implementation and present an extended performance evaluation on GPFS and Lustre parallel file systems. In addition to comparing our methods to traditional approaches, we examine the performance of MPI-IO caching under direct I O mode to bypass the underlying file system cache. We also investigate the performance impact of two file domain partitioning methods to MPI collective I O operations: one which creates a balanced workload and the other which aligns accesses to the file system stripe size. In our experiments, alignment results in better performance by reducing file lock contention. When the cache page size is set to a multiple of the stripe size, MPI-IO caching inherits the same advantage and produces significantly improved I O bandwidth.
@cite_3: A large number of MPI implementations are currently available, each of which emphasize different aspects of high-performance computing or are intended to solve a specific research problem. The result is a myriad of incompatible MPI implementations, all of which require separate installation, and the combination of which present significant logistical challenges for end users. Building upon prior research, and influenced by experience gained from the code bases of the LAM MPI, LA-MPI, and FT-MPI projects, Open MPI is an all-new, production-quality MPI-2 implementation that is fundamentally centered around component concepts. Open MPI provides a unique combination of novel features previously unavailable in an open-source, production-quality implementation of MPI. Its component architecture provides both a stable platform for third-party research as well as enabling the run-time composition of independent software add-ons. This paper presents a high-level overview the goals, design, and implementation of Open MPI.
Many MPI libraries such as MPICH @cite_26 and OpenMPI @cite_3 adopt ROMIO as the implementation for MPI-IO functions. For parallel jobs running multiple MPI processes per compute nodes, ROMIO selects one aggregator per node in its default settings. The selection of I O aggregators is implemented at file open time. If the underneath file system is a Lustre, the default setting changes to select the number of I O aggregators equal to the Lustre file striping count. This strategy produces a ono-to-one mapping between the aggregators and the file servers (OSTs) which avoids any possible file lock conflicts and achieves the best I O performance @cite_21 . When the aggregate file access region is larger than the number of aggregators times the file stripe size, the two-phase I O is carried out in multiple rounds.