Distributed Architecture for FPGA-based Superconducting Qubit Control
Neelay Fruitwala Gang Huang Yilun Xu Abhi Rajagopala Akel Hashim Ravi K. Naik Kasra Nowrouzi David I. Santiago Irfan Siddiqi







Distributed Architecture for FPGA-based Superconducting Qubit Control
    
Neelay Fruitwala,^1
Gang Huang,^1
Yilun Xu,^1
Abhi Rajagopala,^1
Akel Hashim,^1,2
Ravi K. Naik, ^1,2
Kasra Nowrouzi,^1,2
David I. Santiago,^1
and Irfan Siddiqi^1,2


^1Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA

^2University of California at Berkeley, Berkeley, CA 94720, USA
    Received 28 February 2024 / Accepted 23 April 2024
============================================================================================================================================================================================================================================================================================================





 Quantum circuits utilizing real time feedback techniques (such as active reset and mid-circuit measurement) are a powerful tool for NISQ-era quantum computing. Such techniques are crucial for implementing error correction protocols, and can reduce the resource requirements of certain quantum algorithms. Realizing these capabilities requires flexible, low-latency classical control. We have developed a custom FPGA-based processor architecture for QubiC, an open source platform for superconducting qubit control. Our architecture is distributed in nature, and consists of a bank of lightweight cores, each configured to control a small (1-3) number of signal generator channels. Each core is capable of executing parameterized control and readout pulses, as well as performing arbitrary control flow based on mid-circuit measurement results. We have also developed a modular compiler stack and domain-specific intermediate representation for programming the processor. Our representation allows users to specify circuits using both gate and pulse-level abstractions, and includes high-level control flow constructs (e.g. if-else blocks and loops). The compiler stack is designed to integrate with quantum software tools and programming languages, such as TrueQ, pyGSTi, and OpenQASM3. In this work, we will detail the design of both the processor and compiler stack, and demonstrate its capabilities with a quantum state teleportation experiment using transmon qubits at the LBNL Advanced Quantum Testbed. 










§ INTRODUCTION


Room temperature RF control systems have become a critical part of the superconducting quantum computing stack.
With qubit counts in the 10s to 100s, general-purpose RF measurement equipment, such as AWGs (arbitrary waveform generators) combined with discrete RF components, have proven to be overly costly and inefficient for qubit control and measurement. As a result, special-purpose instrumentation has emerged, in both the commercial <cit.> and academic <cit.> realms. These systems integrate pulse sequencing, digital pulse generation, and readout, and are typically built around commercially available FPGAs or SoCs.

The ability to make real-time control decisions based on mid-circuit measurements is becoming an increasingly important part of quantum hardware systems; being a key part of several proposed <cit.> and realized <cit.> quantum algorithms. For superconducting qubits, with coherence times ∼ 100 μ s, realtime feedback requires a controller with latencies ∼ 100 ns. In practice, this means that the feedback control logic must be tightly integrated with the pulse sequencing layer; using an external controller or CPU would significantly increase latency. 

In this work, we present an FPGA-based distributed control architecture which combines pulse sequencing with arbitrary measurement-based control flow. Our design consists of a bank of lightweight, configurable processor cores that are designed to tightly integrate with the puslse generation and signal processing gateware. We also provide a Python/JSON-based intermediate representation for writing and compiling dynamic quantum programs.


 







§ OVERALL APPROACH AND SYSTEM REQUIREMENTS


The scope of this work includes the pulse sequencing and parameterization layer of the FPGA gateware – not the digital pulse generator modules themselves. We designed this layer to interface with the QubiC 2.0 <cit.> pulse generation and readout modules; though we believe that our architecture can be adapted to other qubit control systems that use digital pulse synthesis methods. 

We designed our system around the following principles/requirements:

    
  * Pulse-centric design: the primary control primitives are RF control/demodulation pulses; no intrinsic information or assumptions about quantum (unitary) operations being performed. This simplifies the processor core design and instruction set, and makes it straightforward to implement non-standard unitary operations (e.g. optimal control based approaches and certain calibration sequences<cit.>).
    
  * Low-latency: Superconducting qubits have coherence times ∼ 100 μ s. This means that for conditional operations based on mid-circuit measurements, we require an end-to-end feedback latency ∼ 100 ns. 
    
  * Lightweight: Real-time pulse generation places high demand on FPGA logic and memory resources, particularly on high-channel count devices such as the Gen3 Xilinx RFSoC <cit.>. So, the pulse-sequencing layer should be as lightweight as possible to accommodate a large number of pulse generators on the same SoC/FPGA. 
    
  * Flexible: Superconducting qubit systems have a wide variety of architectures and qubit modalities, each with differing control needs (e.g. readout multiplexing factor, qubit coupler control, and desired instantaneous bandwidth). Our architecture needs to accommodate this variety of pulse generator configurations, and be straightforward to configure at the gateware and software level.




§ ARCHITECTURE


Our architecture consists of a bank of soft processor cores that is responsible for the realtime execution of quantum programs, which involves pulse sequencing, parameterization, and triggering. Each processor core is lightweight, and is designed to interface with a small (∼ 1 - 5) number of digital pulse generators that are used for qubit control and readout. This design mirrors the parallelism inherent to quantum circuit execution, which ensures scalability; having a bank of parallel cores (and fixed number of output channels per core) avoids bottlenecks/latency issues that can arise in single-threaded designs as channel count grows. To enable mid-circuit feedforward operations, we also include an extensible “function processor" module for aggregating and distributing (optionally processed) measurement results to the processor cores. 





 §.§ Processor Core


Each processor core implements a custom instruction set architecture (ISA) consisting of pulse commands for real-time control of the associated signal generators, as well as standard arithmetic and control flow instructions for on-the-fly pulse parameterization and the execution of dynamic quantum programs. The full instruction set is detailed in section <ref>. 



  §.§.§ Signal Generator Interface


Each core is responsible for controlling a small bank of signal generators in real time. This involves both: 1) specifying pulse parameters, such as frequency, phase, and modulation envelope; and 2) triggering the pulse at the correct time. 

QubiC 2.0 <cit.> uses DDS-based (direct digital synthesis) pulse generation modules, which can synthesize a carrier tone at the provided frequency, phase, and amplitude, and can apply a complex modulation envelope given by a time series of values. In the QubiC 2.0 core → signal generator interface, the phase, amplitude, and pulse duration are provided directly via a bus, while the envelope (stored as a series of time-domain values) and frequency (stored as a series of phase offsets per unit time) are pre-allocated in dedicated memory banks, which are configured when uploading the quantum program to the FPGA. It is then the address of the envelope/frequency within these buffers that is specified by the processor core.

The processor core → signal generator interface consists of the following components: 

    
  * Register for storing pulse parameters. Amplitude (16-bit), phase (17-bit), and pulse duration (12-bit) are provided directly, along with pointers to the locations of the modulation envelope (12-bit address) and frequency (9-bit address) in their respective buffers. A configuration word (4-bit) is reserved for miscellaneous parameters.
    
  * 1-bit active high pulse trigger ()


The pulse register fields and trigger time are configured by pulse instructions; see section <ref> for details. 



  §.§.§ Pulse Timing and Synchronization


All pulse triggers are referenced to an internal counter, which is reset at the beginning of the program. This reset is synchronized across all cores. Additionally, QubiC 2.0 has mechanisms for clock sychronization + synchronized reset across multiple FPGA boards <cit.>, ensuring that all pulse triggers and reference clocks are synchronized even when cores are distributed across hardware.



  §.§.§ Microarchitecture



The processor core microarchitecture is outlined in figure <ref>. It is similar to a simple MIPS <cit.> architecture, with a general-purpose 16x 32-bit register bank, 32-bit ALU (arithmetic logic unit), and instruction pointer (or program counter) for interfacing with program memory. For simplicity, the ALU only implements comparison, arithmetic (addition and subtraction), and identity operations. Instructions are implemented using a simple multi-cycle state machine with pipelined instruction fetching. The program memory, pulse interface, and function processor interface are implemented generically in SystemVerilog for portability. We chose an instruction width of 128 bits to accommodate the full 71-bit pulse register, along with the 32-bit pulse start time and other instruction metadata.





 §.§ Function Processor


Each processor core implements a “function processor” interface for connecting to external computational resources. This interface is primarily intended for requesting/receiving (optionally processed) measurement results, although any data/computation with a compatible format can be requested.

The core can request data over this interface by specifying an (implementation specific) 8-bit ID which encodes the type of data to retrieve or computation to perform. Once the data is ready, it is returned as a 32-bit word, along with a ready signal. This request/receive pipeline is triggered by a special instruction, which halts the execution of the core until the resulting data is received. At that point, it can be stored in a register, used as a pulse parameter, or used for a conditional branching decision.

In the current implementation on QubiC 2.0, the function processor interface simply accesses a memory bank containing the most recent state discriminated measurement result from each of the eight qubits driven by the respective FPGA board. This allows any core to request a result from any qubit (provided that it is driven by the same FPGA). Future implementations may extend the function processor to include results from different (synchronized) boards or application-specific measurement decoders.



§ INSTRUCTION SET ARCHITECTURE



Each processor core implements an instruction set consisting of 1) pulse instructions, for parameterizing and triggering pulses; 2) standard register arithmetic and control flow instructions; 3) special-purpose instructions for timing control and interaction with the FPROC interface. In the following section we provide a general overview of the different instruction types; an exhaustive reference can be found in <cit.>.



 §.§ Pulse Instructions



In general, there are two different types of pulse instructions: , which writes to the specified fields of the pulse register, and , which has all of the functionality of , but also triggers the pulse at the specified trigger time. The general format for both of these instructions can be found in figure <ref>.



  §.§.§ 


Pulse register fields can be written to by either an immediate value or a native processor core register (with the exception of a 4-bit configuration word, which must be an immediate). But, only one processor register can be accessed during any given write. So, the  instruction has two additional bits per field: i) , which controls whether to write to that field, and ii) register/immediate select, which controls (if  is high) whether the input value comes from the selected register, or the instruction immediate.



  §.§.§ 


The  instruction adds a 32-bit  field, which activates the pulse trigger at the provided value, which is in units of FPGA clock cycles since program start and is referenced to an internal counter (figure <ref>). Processor core execution is halted until the pulse is triggered.



 §.§ Timing Control


There are certain situations (for example, when looping over a pulse sequence or waiting for a measurement) where the timing-related behavior of a program must be altered. We provide two instructions for this: the  instruction, which will increment the time reference by a signed immediate or register value, and the  instruction, which halts execution of the core until the provided timestamp.



 §.§ Arithmetic and Logical Operations


Register-based arithmetic and boolean operations are performed using  instructions. Supported operations include boolean comparisons (<, >, =), identities, addition, and subtraction, all on 32-bit signed values. Both register-based and instruction immediates are supported. Results are always stored in a register.








 §.§ Control Flow


Any ALU-based boolean comparison can be used to control a jump instruction, which will set the instruction pointer to an arbitrary location in the program memory. Destination addresses must be instruction immediates. Unconditional jumps are also supported.



 §.§ Function Processor


Function processor instructions are used to request/receive data over the FPROC interface. These instructions extend ordinary ALU and control flow instructions, but replace one of the fields with the FPROC result. For example, the  instruction replaces the RHS input of the jump condition with the FPROC result.










§ ASSEMBLY LANGUAGE


We provide a human-readable assembly language that is approximately a one-to-one mapping to the processor core instruction set. The language is formatted as a list of JSON <cit.> strings, with the assembler and associated infrastructure written in the Python programming language. 

The assembly language instruction fields match those of the instruction set with the following exceptions:

    
  * Pulse parameters: all pulse parameters (frequency, amplitude, phase) are provided as floating point values. Frequency and phase are given in SI units, while amplitude is normalized to the DAC full scale. Envelopes are provided as parameterized functions or complex NumPy arrays. Pulse output channels are named, and resolved to gateware/hardware indices during assembly.
    
  * Register names: for readablity, register names are provided as strings, and are resolved into indices during assembly. 
    
  * Register types: for straightforward pulse parameterization, registers are typed as , , or . All operations on  and  type registers are provided in their respective units ( in range [0, 1] for  and radians for ), and are converted to the corresponding pulse-field word during assembly. No conversion is performed with  type registers.
    
  * FPROC ID: function IDs can optionally be specified according to named output channel attributes in the provided channel configuration file. For example, in the program in figure <ref>, the function ID is provided by the  parameter of the  channel.


The assembler takes as input a separate list of instructions for each core, and generates the following outputs: 1) per core program binaries; 2) corresponding set of envelope and frequency buffers. These binaries are stored in a Python dictionary, where they can be loaded by the low-level QubiC driver software into the FPGA BRAM (block-RAM). The assembler is configured using the following:

    
  *  implementation:  is a generic Python class that is implemented separately for each type of firmware signal generator block. It is responsible for converting the provided pulse phases and amplitudes into the correctly formatted words, and computing the frequency and envelope buffers.
    
  * Channel configuration file: this file maps named output channels to firmware channel indices. It may also optionally parameterize the implemented  class.







§ COMPILER TOOLS AND INTERMEDIATE REPRESENTATION


In order to provide users with a high-level format for writing QubiC programs, and to interface with higher-level tools such as TrueQ <cit.>, OpenQASM <cit.>, and PyGSTi <cit.> we provide a custom intermediate representation (QubiC-IR), along with a set of compiler tools for lowering QubiC-IR to distributed processor assembly. 

We designed QubiC-IR to have the following general attributes:

    
  * Multi-level: In order to provide users with a variety of interfacing options (e.g. native-gate level vs pulse level), QubiC-IR operates at multiple abstraction layers. Only a subset of instructions is directly compilable into distributed processor assembly.
    
  * Program flow is single-threaded; the scheduling and compilation tools will parallelize control operations and determine which core(s) need to be targeted by each instruction.
    
  * As with the assembly language, QubiC-IR is primarily represented as JSON; IR lowering and compilation is performed using a Python API


The bulk of the compilation is performed in a series of passes that transform the IR. Once the IR has been sufficiently lowered, a final pass will convert it to distributed processor assembly. The compiler flow is customizable; users can both configure individual passes and specify the set of passes to run.

In the following sections, we give an overview of IR instruction types and associated compiler flows. A full reference can be found at <cit.>.



 §.§ Control Operations: Gates and Pulses


QubiC-IR supports a  instruction that is largely identical to that of the assembly language. We also support a  instruction that allows the program to be written at the native quantum gate level, which can then be resolved into pulses by specifying a calibration file containing the pulse parameters associated with each gate.



 §.§ Classical Variables and Arithmetic


QubiC-IR supports the declaration and manipulation of variables to perform classical computations. Variables are a generalization of assembly language registers; supported operations and allowed datatypes (, , ) are the same. 

However, unlike registers, a variable can be scoped to multiple processor cores, indicating that the variable declaration itself and any manipulations should be duplicated across the relevant cores as register operations. The scope of any variable is specified by the list of hardware output channels it influences (through either control flow operations or direct pulse parameterization). 



 §.§ Virtual-Z Instructions and Phase Tracking


In general, virtual-Z gates are implemented by applying a phase offset to any subsequent control pulses at the specified qubit frequency. QubiC-IR supports a  instruction for this purpose, with two arguments: 1) qubit frequency, and 2) rotation angle (in radians). The qubit frequency can be named (having been previously declared in the program, or defined in the gate calibration file), or anonymous (specified directly using its numerical value).

By default  instructions are resolved in software; the provided phases are applied directly to the relevant control pulses during compilation. However, hardware (i.e. on-FPGA) resolution is also supported; a  directive can be used to bind the phase of all control pulses at a particular frequency to a declared variable. For example, the snippet:

'name': 'declare', 'var': 'q0_phase', 
    'dtype': 'phase',
    
'name': 'bind_phase', 'var': 'q0_phase', 
    'freq': 'Q0.freq'

will result in all control pulses with frequency  to have their  parameter given by the variable . Hardware phase parameterization is required for certain dynamic circuit operations, such as conditional/repeated application of Z-gates.



 §.§ Control Flow


QubiC-IR supports both high-level and low-level (assembly-like) control flow. At the high-level, there are two instructions:  and and . The  instruction functions as a conditional execution (if/else) statement; the instruction contains an ALU conditional operation to evaluate, and  and  code blocks which conditionally execute depending on the result of the conditional. The  and  blocks can contain any valid IR code, including nested control flow. The  instruction consists of an ALU condition, along with  code that executes repeatedly while the condition evaluates to True.

High-level control flow instructions are resolved into lower level  and  instructions. These are identical to their assembly-level counterparts, except, like ALU arithmetic instructions, they can be scoped (hence duplicated) across multiple processor cores.

After all control flow is lowered to assembly-like control flow, another pass will divide the program into basic blocks, along with the associated control-flow graph (CFG). The CFG is then used by subsequent passes, such as virtual-Z phase resolution and scheduling, to track changes in program state across the full program flow.





 §.§ Function Processor


Special instructions are used to request/receive data over the FPROC interface. As with the assembly language, these instructions extend the normal arithmetic and control flow instructions, replacing the RHS ALU input with the function processor data. 

The QubiC-IR infrastructure can resolve channel names into an assembly-compatible format, and add the appropriate delays to ensure that the processor core has enough time to receive and process the measurement results. This is done using an  object and associated compiler pass, which contains a mapping of named FPROC channels to an associated measurement delay and channel ID.



 §.§ Scheduling


QubiC-IR provides two instructions for specifying timing relationships between gates/pulses –  and . The  instruction delays all subsequent pulses on the specified channels by the provided amount. The  instruction is similar to an OpenQASM <cit.> barrier; it aligns the start times of the following pulses to be played on the indicated channels. 


The QubiC compiler has a scheduling pass, which assigns trigger timestamps to all  instructions. Timestamps are determined by taking into account timing constraints (i.e. delays and barriers), pulse length, and instruction execution time. 

Running the scheduler is optional; users are free to directly provide each pulse with a timestamp. In this case, a linter pass is provided to ensure that the pulse schedule satisfies the execution constraints of the processor core(s) (for example, a pulse cannot be triggered during an ALU or branching operation).



§ FPGA IMPLEMENTATION




The processor cores, function processor, and associated interfaces are implemented in Verilog and SystemVerilog. The current implementation is integrated with QubiC 2.0 on the ZCU216 RFSoC platform. Our design is modular; SystemVerilog interfaces are used to connect the processor cores to the QubiC signal generator, memory banks, and measurement modules. 

A variety of QubiC 2.0 implementations exist, ranging from 4x-16x DAC drive channels, and a variety of signal generator and readout multiplexing configurations <cit.>. In the following sections, we describe a distributed architecture + QubiC 2.0 implementation designed for the AQT Trailblazer QPU (quantum processing unit), which is an 8-qubit superconducting transmon system, with fixed qubit frequency, fixed coupling, and 8x multiplexed readout <cit.>. 

This implementation has eight distributed processor cores; i.e. one for each qubit. Each core controls three signal generator channels (figure <ref>), one for qubit drive (which goes to a dedicated DAC), another for readout drive (which goes to a common multiplexed readout DAC), and another for readout demodulation (which mixes with the readout resonator response tone from the multiplexed readout ADC). All processor cores, signal generator blocks, and readout demodulation are on the same 500 MHz clock domain. All drive and readout DACs are configured to operate at 8 GSPS (gigasamples per second), and the ADCs at 2 GSPS.



 §.§ Resource Utilization






The FPGA resource requirements for our implementation are reported in figure <ref> and can be visualized on the floorplan (figure <ref>). The logic resource requirements (CLB and DSP) of each processor core are minimal in comparison to the resources required by the corresponding signal processing blocks (i.e. control/readout signal generation and readout demodulation for a single qubit), ensuring that our architecture is unlikely to present a significant scaling bottleneck. The BRAM (block RAM) utilization is largely arbitrary, and depends on the desired circuit depth/program size. In our implementation, single-core BRAM utilization is minimal at 0.2 %, corresponding to a program memory capable of storing 2048 128-bit instruction words.



§ EXPERIMENTAL DEMONSTRATION: QUANTUM TELEPORTATION


In order to demonstrate the mid-circuit measurement and feedforward capabilities of our architecture, we performed a quantum state teleportation experiment <cit.>. For this experiment, we used the AQT (Advanced Quantum Testbed) Trailblazer QPU, which has 8 fixed-frequency transmon qubits with linear connectivity <cit.>. Our teleportation circuit is given in figure <ref>; we used the BQSKit compiler <cit.> to translate this circuit into the AQT Trailblazer's native gate set (comprised of X_90, CZ, and virtual-Z(θ) gates).



We performed the teleportation experiment for four different initial states on qubit Q0: | 0 ⟩, | + ⟩, | - ⟩ and | 1 ⟩. The corresponding Z-basis measurement results for qubit Q2 are shown in figure <ref> for the |0 ⟩ and |1 ⟩ initial Q0 states. For the | + ⟩ and | - ⟩ states, we also performed measurements in the X and Y bases to determine the position of the state vector in the X-Y plane of the Bloch sphere (figure <ref>).







A table of measured expectation values for the destination qubit Q2 is given in figure <ref>. For the | 0 ⟩, | + ⟩, and | 1 ⟩ states, the measured expectation values show good agreement with theoretical predictions, indicating successful teleportation of the quantum state from Q0 to Q2. For the | - ⟩ state, the measured expectation values show partial agreement, with significant deviation in the Y-basis. This discrepancy is likely due to a combination of dephasing and readout-induced crosstalk, which are known issues when implementing mid-circuit feedforward operations <cit.>.









§ CONCLUSION


We have developed an open source FPGA-based architecture for superconducting qubit control and measurement. Our architecture supports the execution of dynamic circuits, including mid-circuit measurement and feedforward, and realtime parameter updates. We also provide a modular compiler stack and intermediate representation that supports a variety of abstraction levels and can integrate with standard quantum programming tools.

Our architecture is deployed on the QubiC 2.0 <cit.> system, which currently uses the Xilinx ZCU216 RFSoC evaluation board, and has been used to control the 8-qubit Trailblazer QPU at the LBNL AQT. In addition to the state teleportation demonstration presented in this paper, our system has enabled the demonstration of novel scientific results, including randomized compiling for mid-circuit measurement <cit.>, and measurement-based entanglement generation <cit.>.

Our design and compiler stack is fully open source, and can be found on Gitlab: <https://gitlab.com/LBL-QubiC/distributed_processor>. 








§ ACKNOWLEDGMENT


This work was supported by the U.S. Department of Energy, Office of Science, Advanced Scientific Computing Research Testbeds for Science program, the National Quantum Information Science Research Centers Quantum Systems Accelerator, and the High Energy Physics QUANTISED program under Contract No. DE-AC02-05CH11231.






























IEEEtran













































