Enabling Pulse-level Programming, Compilation, and Execution in XACC
Thien Nguyen Alexander McCaskey
quant-ph physics.comp-ph















  Computer Science and Mathematics 
 Oak Ridge National Laboratory





nguyentm@ornl.gov



  Computer Science and Mathematics 
 Oak Ridge National Laboratory




mccaskeyaj@ornl.gov






Noisy gate-model quantum processing units (QPUs) are currently available from vendors 
over the cloud, and digital quantum programming approaches exist to run low-depth circuits on physical hardware. These 
digital representations are ultimately lowered to pulse-level instructions by vendor quantum control 
systems to affect unitary evolution representative of the submitted digital circuit. 
Vendors are beginning to open this pulse-level control system to the public via specified interfaces. 
Robust programming methodologies, software frameworks, and backend simulation technologies for 
this analog model of quantum computation will prove critical to advancing pulse-level control research and development. 
Prototypical use cases for this include error mitigation, optimal pulse control, and physics-inspired pulse construction. 
Here we present an extension to the XACC quantum-classical software framework that enables 
pulse-level programming for superconducting, gate-model quantum computers, and a novel, general, 
and extensible pulse-level simulation backend for XACC that scales on classical compute clusters via MPI. 
Our work enables custom backend Hamiltonian definitions and gate-level compilation to available 
pulses with a focus on performance and scalability. We end with a demonstration of this capability, and 
show how to use XACC for pertinent pulse-level programming tasks.




































































Enabling Pulse-level Programming, Compilation, and Execution in XACC
    Alexander McCaskey
    Received: date / Accepted: date
====================================================================





§ INTRODUCTION











The emergence of noisy intermediate scale quantum (NISQ) <cit.> processing units and their accessibility to programmers via cloud services in recent years have led to major advances in software frameworks and programming models for quantum-classical computing. For gate-model quantum computing the majority of the programming framework approaches have focused on digital assembly submission to dedicated quantum job servers. This has led to a number of demonstrations of small-scale experiments in machine learning, quantum chemistry, nuclear physics, and quantum field theory <cit.>. However, in the NISQ-era, much stands to be gained from thinking about programming quantum hardware closer to the physical model of the system, specifically, programming at an analog, or pulse-instruction, level. Early results in this regard have demonstrated the utility of pulse-level control in areas such as error mitigation and optimal quantum control <cit.>. Moreover, vendors are starting to propose and implement specifications for pulse-level job submission, opening up this novel, analog programming model to the community at large. In the near-term, programming models and software framework implementations will prove critical in enabling the benefits of low-level analog quantum control and programming. Moreover, robust simulation technologies are required to promote pulse-level research productivity due to the scarce and remote characteristic of available pulse-level physical backends. 

Programming mechanisms for pulse-level quantum computation is very much at an early research and development phase. IBM has recently put forward the OpenPulse specification, with a corresponding early implementation in the Qiskit-Terra framework <cit.>. This implementation provides a high-level Pythonic approach for pulse-level program composition and specifically targets IBM physical backends. Pulse-level simulation tools exist but are available in an ad-hoc manner - most users leverage frameworks such as QuTip for quick prototyping and scripting <cit.>. These approaches lack integration with physical hardware backends and internal circuit representations for the typical quantum programming, compilation, and execution workflow. There is a need for a system-level programming, compilation, and execution workflow that enables scalable simulation on available distributed-memory compute architectures. 

  
Here we detail our solution to the current lack of programming tools and simulation technologies for pulse-level programming of gate-model quantum architectures. We leverage the system-level XACC heterogeneous quantum-classical software framework to provide a programming mechanism and associated compilation tooling for pulse-level quantum control systems. Initially we target the OpenPulse specification, but believe that the hardware-agnostic characteristic of XACC will enable us to extend this model for future quantum control specifications (such as Rigetti Quilt). We put forward a state-of-the-art pulse-level simulation backend for XACC that builds off the QuaC (Quantum in C) open quantum system dynamics solver. QuaC builds off of the PETSc matrix algebra library, and therefore our work immediately scales on distributed memory architectures via the Message Passing Interface (MPI). We detail programming, compilation, and execution layers of this overall pulse-level programming model, and provide detailed demonstrations exhibiting the use of this API. Our goal with this work is to provide a unique pulse-level programming capability with a robust simulation backend to enable research productivity for tasks in error mitigation, physics-inspired pulse learning, and quantum optimal control. 

    

The paper is organized as follows. In Section <ref>, we review some key features and components
of the XACC framework. Next, in <ref>, we briefly summarize the OpenPulse specification which we adopt
in this work. Section <ref> describes the overall software architecture of our pulse-level extension
and how it fits in with the framework. Usage demonstrations and conclusions are given in Section <ref>
and Section <ref>, respectively.



§ XACC FRAMEWORK
 

Recent progress in the development of programmable quantum computing architectures has resulted in a number of programming models and associated software stack implementations enabling digital gate-level programming of these nascent architectures. Most efforts have been put forward by hardware vendors and typically only target the backend architecture the vendor provides. Most of these efforts are provided via high-level Pythonic circuit construction data structures, with a few specifically extending existing classical languages for heterogeneous quantum-classical computing. This landscape has necessitated the development of unified, system-level architectures for robust integration efforts across quantum architectures, languages, and compiler techniques <cit.>. 


Recently, the XACC system-level software framework has been developed to address these requirements. XACC provides an extensible, holistic framework for quantum-classical computing in C++ in a way that is agnostic to the backend quantum hardware type as well as front-end language transpilation and compilation <cit.>. 
At its core, XACC provides a unified and simplified quantum programming model
that can cross hardware boundaries while not compromising efficiency in terms of quantum hardware 
utilization or developers productivity. Moreover, the system-level language approach ensures performance of classical processing and support code which
is essential in the NISQ-era where hybrid algorithms are one of most viable approaches that can 
provide substantial quantum advantage.

As a comprehensive framework, XACC (see Fig. <ref>) 
adheres to the newly proposed QCOR language extension specification <cit.>, which provides guidance on the programming and memory model,
support data structures, and the execution model. XACC provides a reference implementation that is open-source, fully-functional, actively being maintained and developed <cit.>.
The base implementation supports all major quantum assembly dialects, e.g. Open QASM, Quil, and Staq <cit.>, as well as its own 
XASM dialect. XACC has back-end connections, e.g. via RESTful APIs, to hardware platforms provided by IBM, Rigetti, IonQ, and Dwave, thereby enabling a write-once-deploy/test-everywhere capability to programmers and researchers. Another key focus area is the integration of quantum simulation software to use as drop-in replacements for real hardware back-end during development.
XACC support simulation backends from the IBM Terra/AER package <cit.>, the Quantum library <cit.>, the ITensor library <cit.>, and the high-performance ExaTN tensor library <cit.>.
This level of modularity and extensibility, and the overall strong focus on performance, significantly enhances developer productivity.

XACC provides an extensive set of libraries that are constantly updated. This includes algorithm implementations, from simple static algorithms,
e.g. quantum Fourier transform, reversible classical computing, to novel quantum variational ansatz circuits, such as UCCSD, ASWAP, 
or quantum circuit structure learning. Observables, such as Pauli or spin operators, are a native construct within the framework, and enable XACC to
automatically inject gates and measurement operations in order to compute pertinent expectation values. Moreover, XACC supports 
common hybrid algorithms like the variational quantum eigensolver and data-driven circuit learning routines, as well as characterization 
tools such as quantum process tomography. Again, these libraries are meant to provide developers
with drop-in, backend-compatible, and fully-validated solutions without needing them to implement or maintain the code. 

Lastly, XACC provides user-friendly Python language bindings via Pybind11 <cit.>. This is not a specification-driven feature but is equally important since
C++, despite being highly performant, requires substantial familiarity with professional software development procedures,
e.g., building and linking, that may represent a barrier to domain computational scientists. 
Thanks to the language bindings, users can use XACC services from their Python shell and, more importantly, can provide
implementations to framework constructs, e.g. Accelerators, Compilers, Algorithms, etc., natively from Python while adhering to
the common interfaces that have been exposed via the language bindings.








§ OPENPULSE SPECIFICATION
 

The OpenPulse specification <cit.> was proposed by IBM as an open and implementation agnostic standard for pulse-level control. The specification defines JSON schemas for system Hamiltonians, drive and control channel
configurations, pulses, and pulse-sequence definitions. Since this is the first pulse-level standard put forward in the industry, our XACC implementation adheres to the OpenPulse specification.



 §.§ Hamiltonian
 
The specification defines a machine-readable encoding scheme for the system Hamiltonian as summarized in Table <ref>.
Specifically, the Hamiltonian field (h_str in the OpenPulse JSON object) is a list of strings describing various terms in the Hamiltonian.
These terms can be time-independent (control-free evolution) or time-dependent (driven by an input channel). 
The latter case is denoted by the characters || followed by a channel name 
(either a drive channel, Di, or a control channel, Ui).

Also, there is a set of pre-defined operators, such as Pauli operators (X, Y, or Z) 
or ladder operators (e.g. SM denotes the spin σ_- or the photon destroy (a) operator), that can be used to construct those Hamiltonian terms. For brevity, the standard also allows running summation terms with integer indices to group terms that have the same structure. 

All variables in the Hamiltonian expressions must have corresponding assignments in the vars map to resolve the Hamiltonian dynamics. Additionally, qubit subsystems can be modeled as higher-dimensional systems (e.g. qutrits or qudits) by specifying their dimensions in the qub field.



 §.§ Pulses and Commands


Pulses (waveforms of drive signals) are defined as sequences of sample points (complex numbers). They are
specified as a JSON object with a name and a samples field (see Fig. <ref>).










A standard OpenPulse back-end configuration often contains a calibrated pulse library and a map from quantum gates to sequences of pulses. Each mapping entry is called a command definition, or cmd-def. A sample cmd-def JSON entry is also shown in Fig. <ref>. 


As we can see, at the pulse level, a digital quantum gate is decomposed into a sequence of pulses (referred to by name)
and frame changes (changing the local oscillator's phase) on specific channels at specific time.
The pulse sequence for each gate is defined independently referencing the gate start time as time zero. 
A back-end pulse library can implement an arbitrary set of universal gates and hence it is up to
the high-level gate compiler to perform any necessary transformations and mappings.



§ SOFTWARE ARCHITECTURE
 

The extensible, layered and future-proof interfaces that the XACC framework implements provide a foundation upon which our pulse-level extension is built. 
We supplement the front-end intermediate representation with a pulse instruction extension and implement a middle-end pipeline to enable gate-to-pulse translations. The processed pulse-level program can then be executed on the QuaC simulator/accelerator backend which performs a fully dynamical simulation of the quantum system driven by those pulse instructions. Of course, with backend support for quantum hardware from IBM, XACC can take this pulse-level IR and execute it on physical architectures as well.



 §.§ Pulse IR


In the XACC framework, the Instruction and 
CompositeInstruction intermediate representation (IR) interfaces are the building block of in-memory data structures capturing the semantics of a user-specified quantum program. 

In a usual setting, an Instruction represents a quantum kernel instruction, i.e. a quantum gate. To handle analog-like pulse instructions, we define a Pulse sub-class of Instruction (Fig. <ref>  fig:instruction_polymorphism) 
that encapsulates extra metadata associated with analog control such as data samples, drive channel and pulse timing. We extend Instruction with typical getter and setter methods for this metadata, which by default (for digital gate instructions) has no effect. We implement these methods on Pulse to provide this metadata upon request. Moreover, we extend Instruction with a isAnalog() method to easily indicate if a set of  Instructions are digital or analog. 
This allows us to construct flexible combinations of instructions (analog or digital) when building up a CompositeInstruction, i.e. a quantum circuit/kernel.   


In pulse mode, analog IR objects can be constructed via direct calls to the IR factory methods (Fig. <ref> & <ref>) or using XACC pulse library loader (Fig. <ref>.)
The former offers maximal flexibility whereby arbitrary pulse instructions can be constructed from sample arrays. The latter provides a means for users to utilize any OpenPulse-conformed pulse library to build up their pulse program in XACC. Pre-defined and calibrated pulses are loaded into the instruction registry of the XACC IR provider for later use by referring to the pulse instruction name. In XACC, pulse composites that implement gates are prefixed with a 'pulse::' phrase. It is worth noting that, in pulse execution mode, XACC will automatically lower digital gate instructions to pulse composites as described in Section <ref>.


Besides being able to construct quantum kernels from pulse-level IR objects, the framework has extended its compilation infrastructure to handle automatic gate-to-pulse IR lowering. Hence, existing quantum circuits can be translated into pulse sequences and simulated dynamically within the framework itself. The existing XACC CompositeInstruction API allows us to add, remove, or replace component Instruction nodes, which is crucial for the pulse-aware lowering of quantum kernels as we will describe in detail next.



 §.§ Pulse Program Compilation

An important feature that we want to enable for XACC is the ability to transpile or compile gate or assembly level descriptions of quantum circuits into analog pulses that have been arranged to individual channels at specific time slots. This capability previously is only available on non-public hardware vendor backend software stacks. Since we want to implement an end-to-end pulse-capable stack for XACC that spans from front-end to back-end, including a pulse emulator backend that we will introduce in Section <ref>, we also implement a generic digital quantum gate to pulse compilation workflow.

Starting with an CompositeInstruction tree structure, e.g. by compiling a quantum assembly program, XACC provides a flexible InstructionVisitor interface to traverse this Instruction tree and apply custom operations for each concrete Instruction (gate) type. Following this visitor pattern, we have implemented a two-pass gate-to-pulse lowering and scheduling procedure as described in Algorithm <ref> & <ref>. 



In the first pass, all gates are transpiled into pulses or pulse sequences via Instruction registry look-up. If a gate has a direct pulse command definition available, e.g. X or CNOT gates, it will be substituted by the associated pulse sequence. However, many quantum gates do not have direct pulse definitions, e.g. due to extended over-complete instruction sets. In that case, we rely on the universal instruction set that the registered pulse library implemented to decompose the gate into its universal gate equivalent and then lower those decomposed gates into pulses. After converting the original gate IR into universal gate set IR's, we revisit those IR nodes which should then have pulse command definitions required for the lowering. 

It is worth noting that pulse sequences are time-domain series all referencing time zero. These pulse sequences are then assembled as blocks in order to implement a specific quantum circuit. Therefore we must implement a compiler pass that schedules pulses globally. 

This high-level pulse scheduling is described in Algorithm <ref>. To achieve this, we have extended XACC with a new Scheduler service interface which we implement specifically for global pulse scheduling - the PulseScheduler. 

This implementation basically block-shifts pulse-level CompositeInstructions, i.e. those pulse sequences that implement quantum gates, by adjusting the start time of component pulses. This maintains the relative timing of pulses within a pulse sequence as well as the atomicity of quantum gates once they are converted into pulses. 

At the end of this procedure, we have a globally scheduled deck of pulse instructions, e.g. the one listed in Fig. <ref>, dictating analog signals on channels driving the system Hamiltonian. This pulse program can hence be submitted to a pulse-capable hardware backend or to the QuaC simulator backend that we have provided as part of the XACC suite.




 §.§ QuaC Pulse Accelerator
 
The XACC framework defines an Accelerator service 
that provides a unified interface for all framework services
that need access to a quantum computer, either
via a remote REST connection or local simulator. 
Therefore, we also implement our pulse-level simulator to adhere to this Accelerator
interface in order to guarantee drop-in compatibility with existing services as illustrated in Fig. <ref>. In other words, existing projects 
written for the framework, e.g. for a quantum chemistry problem or a some other quantum algorithm, can immediately
utilize the pulse-capable simulator back-end without any modifications.


The overall architecture of the simulator is illustrated in Fig. <ref>. Unlike other circuit-based back-ends,
a pulse-level back-end must be aware of the quantum system dynamics, i.e. the Hamiltonian, and also be able to generate continuous time-dependent signals to drive the Hamiltonian according to the input gate or pulse instructions. In the following sub-sections, we will describe these pulse-specific components in greater detail.



  §.§.§ Hamiltonian Parser
 
As previously mentioned, we adopt the OpenPulse specification, which
standardizes various data formats for pulse-level control. With respect to the quantum hardware system dynamics,
as summarized in Section <ref>, the Hamiltonian is expressed as a list of terms,
each of which can be time-independent, time-dependent (driven by a channel) or an indexed sum of 
similar terms. 

C++ polymorphism provides an effective way to capture the Hamiltonian abstraction as depicted in Fig. <ref>
whereby different derived sub-classes of a common abstract class handle concrete types of Hamiltonian terms.
Custom actions on the system dynamics which are specific to each type of term are encapsulated in an apply() method
whose implementation is provided by each child class. Based on the textual signature of each input term, the parser will construct an
appropriate object capturing all information specific to the type of the term. The apply() action will then reconstruct
the dynamics within the simulator, i.e. the Hamiltonian matrix.




  §.§.§ Pulse Controller
 
As depicted in Fig. <ref>, instantaneous values of all driving time-dependent terms are
provided by a centralized ChannelController whose task is to process the scheduled pulse program
to figure out the driving signal values on all channels. This involves (1) tracking which pulse is active on a given 
channel at a given time and how far along its data sample list we are, (2) accounting for all frame changes (local oscillator phase manipulation) 
that have requested on any given channels, and (3) performing LO-mixing to compute the absolute time signals. The way in which
our pulse controller handles these tasks is summarized in Algorithm <ref>.   



  §.§.§ QuaC Simulator
 
The numerical back-end of our implementation, so-called Quantum in C - QuaC <cit.>, as shown in Fig. <ref> 
is a quantum dynamical solver based on the Portable, Extensible Toolkit for Scientific Computation (PETSc) library <cit.>.
Built upon PETSc's scalable (parallel) data-structures and runtimes, the QuaC back-end in XACC supports multi-node
parallelism via the industry-standard Message Passing Interface (MPI), which promises extended scalability across classical computing infrastructure.

As a standalone package, QuaC provides a flexible open quantum systems simulator capable of solving generic Lindblad master equations (<ref>) as well as applying gate-type operators. 

    dρ/dt   = -i[H, ρ] + ∑_k γ_k (A_k ρ A_k^† - 1/2A_k^† A_k ρ  - 1/2ρ A_k^† A_k) 
       = ℒ(ρ)


The vectorized density matrix (ρ) and the superoperator (ℒ) representing the right-hand side of the master equation are modeled by the abstract PETSc vector (Vec) and matrix (Mat) objects whose implementation details, such as memory allocation and storage across multiple MPI processes, are handled implicitly by PETSc for maximal portability. Specifically, the superoperator matrix is stored in the compressed sparse row (CSR) format across multiple (parallel) processes using the PETSc-builtin MPI AIJ type.

Qubit sub-system operators, e.g. Pauli operators, which can be used in the OpenPulse Hamiltonian, are implicitly modeled internally as functions that retrieve non-zero matrix elements and indices to take advantage of their sparsity. These functions are used to project operators' local elements into the overall RHS superoperator matrix when we process Hamiltonian terms, i.e. executing the apply() method (Fig. <ref>) on each term. All channel-driven terms in the Hamiltonian are modeled as a time-dependent function to compute the Jacobian which is a PETSc callback at each time step.

It is worth noting that PETSc offers a wide variety of time integrators, such as forward and backward Euler methods, symplectic methods, backward differentiation formula (BDF) method, etc. By default, QuaC uses TSRK3BS integrator type (third-order Runge-Kutta scheme of Bogacki-Shampine with 2nd order embedded method) when solving the master equation. 

As depicted in Fig. <ref>, all these low-level numerical computation details are encapsulated within the xacc::Accelerator interface thanks to the high-level system model, pulse channel controller, and accelerator buffer data structures. The first and second objects translate the input Hamiltonian, backend configurations (e.g. LO frequencies), and the pulse program into QuaC and PETSc data structures and functions required for solving the pulse-driven system dynamics. The third object (a core abstraction from XACC modeling a register of qubits and exposing pertinent backend-execution results) captures all necessary outputs from the simulation such as the probability distribution which can be used to generate a measurement shot count distribution.



  §.§.§ Pulse Generator Utility
 
In an effort to improve the usability and ease of integration, we also put forward a set of utility functions that generate pulse data series for common pulse shapes, such as square or Gaussian pulses. Given the analytical parameters of the envelope function, e.g. the width (variance) parameter of a Gaussian function, the sampling time (dt), and the pulse duration, these utilities will generate the corresponding array of data samples. In the most generic form, users can also supply a string-based functional expression which will then be parsed and evaluated to compute the time-domain samples.

These pulse generation utilities are also made available to the Python language binding for Pythonic programming. Python has a rich collection of signal processing libraries, e.g. scipy.signal, which can be used independently or in conjunction with these XACC-provided facilities to construct input pulses. 



§ DEMONSTRATION
 
In this section, we seek to demonstrate the usage of XACC pulse-level application programming interfaces to accomplish tasks ranging from simple physics experiments to complex, cross-service applications involving integration of new and existing XACC components.



 §.§ Arbitrary pulse definition


As a first example, we describe how arbitrary pulse waveforms can be described via direct construction and manipulation of pulse-level IR instances (see Fig. <ref>).
In particular, the setSamples method of the Pulse class (see also Fig. <ref>fig:instruction_polymorphism) 
can be used to attach a list of sample data points to a pulse IR. The xacc::getAccelerator API function enables one to retrieve an Accelerator
back-end by name from the framework service registry. The CompositeInstruction, i.e. quantum program,
which contains pulse-level instructions can be simulated/executed on the back-end by invoking the execute method and passing references to the 
program and the AccelerationBuffer (see Fig. <ref>) to which the results are persisted.





 §.§ Finding resonance frequencies


In this example, we demonstrate the way the system Hamiltonian is specified (Fig. <ref>) and
how to configure various simulation parameters to experiment on- and off-resonance driving of qubits.

  

Specifically, we will apply an X gate (in XASM dialect, Fig. <ref>), 
which will be translated to a corresponding pulse, on each qubit. Since the X-gate pulse will actually be mixed with the LO signal before being applied to the qubit,
we can vary the LO frequency in order to find the qubit resonance frequency.
In the below simulation, we sweep the frequency of the local oscillators of drive channel 0 (line 20, Fig. <ref>),  
and drive channel 1 (line 45, Fig. <ref>), where the X-gate pulses are applied and collect the probability that
the corresponding qubit is measured in the |1⟩ state (line 42, Fig. <ref>.) The results are shown in Fig. <ref> which, as expected,
verify the resonance characteristics of the back-end Hamiltonian.



This example also demonstrates the end-to-end gate-to-pulse transformation capabilities of the XACC framework. 
In particular, rather than needing users to manually specify the pulses, which they can certainly do as in <ref>, 
digital gates are automatically lowered into pulses and dynamically simulated once the QuaC pulse back-end is requested. 
In other words, other quantum assembly language based circuits can also be simulated at the pulse level targeting a
specific hardware model via the QuaC simulator back-end.

