\subsubsection{\stid{2.01} \tools\ Software Development Kits} 

\paragraph{Overview}
The Software Development Tools SDK is a collection of independent projects specifically targeted to address performance analysis at scale. The primary responsibility of the SDK is to coordinate the disparate development, testing, and deployment activities of many individual projects to produce a unified set of tools ready for use on the upcoming exascale machines. The efforts in support of the SDK are designed to fit within the overarching goal to leverage and integrate data measurement, acquisition, storage, analysis, and visualization techniques being developed across the ECP ST ecosystem.


\paragraph{Key Challenges}
In addition to the general challenges faced by all of the SDKs outlined in Section~\ref{subsubsect:ecosystem-sdk}, the unique position of the \tools\ SDK between the hardware teams and the application developers requires additional effort in preparing today’s software to run on yet-unknown architectures and runtimes to be delivered by the end of ECP.

\paragraph{Solution Strategy}
The primary mechanism for mitigating risk in the SDK is the \textit{Readiness Survey}. This survey is designed to assess the current status of each product in the SDK in six key areas: software availability, documentation, testing, Spack build support, SDK integration, and path forward technology utilization. By periodically assessing the progress of the individual L4 products in the SDK, we will use the survey to identify and resolve current hardware architecture dependencies, plan for future architecture changes, and increase adoption of the Continuous Integration (CI) testing workflow to reduce this risk.

Critically, the survey will allow us to accomplish this by providing a direct communication channel between the SDK maintainers and the L4 product developers allowing us to identify current architecture dependencies in each project and compare them with existing and emerging ECP platforms. Our initial efforts will be to increase support for today’s heterogeneous CPU architectures across the DOE facilities (e.g., x86, Power, ARM, etc.) to ensure a minimum level of usability on these platforms. We will then focus on current accelerator architectures- namely GPGPU computing. As new architectures arise, we will re-issue the survey and use this same process to provide guidance to the L4 product as they develop support for them.

The survey also allows us to monitor the increased adoption of the proposed ECP CI testing workflow. This will be crucial to understanding each project’s interoperability with not only the other projects within the Tools SDK, but all applications across the ECP Software Technologies landscape. Additionally, it will serve as a bridge between the HI teams working with the facilities and the software teams working across the SDK. By relaying new hardware requirements from the facilities to the software developers, we can closely monitor support for both new and existing systems. Conversely, giving feedback to the facilities regarding compiler support and buildability of library dependencies will guide software adoption on those platforms.

\paragraph{Recent Progress}
The Readiness Survey was re-issued to each L4 product in September 2021. There have been no changes to planned GPU support. Work is actively ongoing to get the tools operating with the Intel GPUs for Aurora via the Arcticus early access system and the AMD GPUs for Frontier via the Spock early access system. In Q321, NERSC enabled ECP users to evaluate their preliminary version of Perlmutter. Only two of the tools L4 projects are targeting work for that system, currently. We expect that number to grow in the next few months.

Support for automated testing remains a challenge area that all of the projects are aware of and will continue to be a point of focus for the SDK. Thus far, two of the six products, Dyninst and TAU, have had successful runs of their complete test suites on pre-exascale systems. With this work, both products now have working tests that can be employed through scriptable executions. This represents an essential component of software sustainability to demonstrate and track correctness in the presence of code changes for these products. Additionally, all of the L4 projects have both an entry in the E4S test suite as well as 'smoke test' functionality in their Spack packages. These testing modalities will be automatically executed by the E4S team when assessing releases. The Spack smoke tests are also executed by the Spack CI pipeline when evaluating pull requests for a package as well as part of preparation for a Spack release.

Continuous Integration (CI) testing remains a still-larger challenge for the SDK. This is due in part to some products not having scriptable testing capabilities and also in part to more general challenges of using CI at the facilities. With the decommissioning of the OSTI Gitlab, we have moved to a distributed model of running CI at each facility separately. We currently have a preliminary CI pipeline that can be manually executed to build the entire SDK.

Starting in Q122, the SDK will be expanded from six to thirteen L4 projects. This was the result of a request from LLNL affiliates to improve support and utilization of their internally-developed tools. These changes were introduced too late to be explicitly included in the FY22 P6 activities, but they will be incorporated into the SDK's milestone efforts, nonetheless.

The Tools SDK has been an active contributor to developing the E4S Community Policies (Figure \ref{fig:E4S-Community-Policies-V1}). From the FY21 \textit{Readiness Survey} results, two of the L4 projects are planning to be member packages in E4S. They are working toward compatibility in FY22 and beyond. It should be noted that these policies are not a requirement for inclusion in the SDK. Instead, they are good-faith efforts to demonstrate how software in the sciences can be made more accessible and portable.

\paragraph{Next Steps}
In FY21, we continued our efforts to assess builds with multiple compilers on early access systems. Results from these tests were fed back into the L4 products to guide development of spack packages, bug/issue-reporting workflows, and integration into the greater ECP software ecosystem. As the number of early access systems increases in FY22, we expect this work to remain central to our efforts. Additionally, we want more of the L4 members to be consistently built under the auspices of the SDK development efforts as an addition to the various builds carried out by the Spack and E4S developers. A large portion of our SDK work allocation in FY22 will be dedicated to this task with an emphasis on utilizing Continuous Integration at the various labs as a vehicle.
